[IQUG] IQ table corruption

John Anthony chirayithaj at gmail.com
Thu Oct 3 09:26:17 MST 2019


Amoolya,

You are already working with the best!!

Have great respect for the guys you are working with but experience also
tells me that IQ does not recover from a massive disk corruption,
especially at the scale you are at. I speculate the the details of the
underlying data structures make this a difficult computational problem,
else the guys would have figured this out.

If anything, the corruption spreads over time. Is why i recommend starting
with another IQ server, copying the data, difficult as that is. And this
time, take frequent backups, retain backups over longer duration, and
perform ongoing consistency checks, even when they take very long, to
ensure this does not happen again!

The presentation i was supposed to make to your team started with "You will
Never Recover from a Storage Failure". I have been there!

IQ needs to , at PB scale, be more accepting of storage failure, have
redundancy as fallback. Ideas behind Hadoop!

-JA

On Wed, Oct 2, 2019 at 9:41 PM Amoolya Ajmera <Amoolya.Ajmera at rci.rogers.com>
wrote:

> @John, can you help us to make it evident from logs about the storage
> being faulty if IQ msg logs are share.  Can you please share your analysis
> for the below log line
>
>
>
> I. 10/01 12:17:16. 0000169027 {1315} main Bufman: Decompression error
> during Read; from slib/s_buf.cxx:1820 into buffer 4d718868
>
>
>
> @Mark we already have a case opened high priority today afternoon and
> there is no action on it yet
>
>
>
> 606489 / 2019 Error while read
>
>
>
>
>
>
>
> *From:* John Anthony <chirayithaj at gmail.com>
> *Sent:* October 2, 2019 7:10 PM
> *To:* Mark Mumy <markdmumy at gmail.com>
> *Cc:* Amoolya Ajmera <Amoolya.Ajmera at rci.rogers.com>; iqug at iqug.org
> *Subject:* Re: [IQUG] IQ table corruption
>
>
>
> *CAUTION*: This email is from an external sender. Do not click links,
> open attachments or reply unless you recognize the sender's email address
> and know the content is safe.
> *ATTENTION* : Ceci vient d’un expéditeur externe. Ne cliquez pas sur les
> liens, n'ouvrez pas les fichiers et ne répondez pas sauf si vous
> reconnaissez l'adresse et êtes sûr que le contenu est sécuritaire.
>
>
>
> Amoolya - because I know what happened and the process you’ll went
> through, and having seen this happen at the kind of scale you have , my
> advise would be to build another IQ instance and copy the data out from the
> original IQ, while keeping the system running via proxy tables.
>
>
>
> It is tough but bad storage , when at scale, messes up IQ!
>
>
>
> -JA
>
>
>
>
>
> On Wed, Oct 2, 2019 at 5:54 PM Mark Mumy <markdmumy at gmail.com> wrote:
>
> Open a case so support can get you the right tools.  And most importantly
> let you know if the issues are already fixed in a newer version.
>
> Mark
>
>
>
> ========================
>
>
>
> Sent from my mobile device
>
>
>
> On Oct 2, 2019, at 15:27, Amoolya Ajmera <Amoolya.Ajmera at rci.rogers.com>
> wrote:
>
> 
>
> We are facing table corruption every now and than on IQ, we are on latest
> 16.0 Pl23
>
>
>
> Anything that can help to track how corruption is going on and what is the
> possible reason behind it that we can track in details from logs in details
>
>
>
>
>
>
>
>
> ------------------------------
>
> This communication is confidential. We only send and receive email on the
> basis of the terms set out at www.rogers.com/web/content/emailnotice
>
>
>
> Ce message est confidentiel. Notre transmission et réception de courriels
> se fait strictement suivant les modalités énoncées dans l’avis publié à www.rogers.com/aviscourriel
>
> ------------------------------
>
> _______________________________________________
> IQUG mailing list
> IQUG at iqug.org
> http://iqug.org/mailman/listinfo/iqug
>
> _______________________________________________
> IQUG mailing list
> IQUG at iqug.org
> http://iqug.org/mailman/listinfo/iqug
>
> --
>
> Sent from mobile device
>
>
>
>
> ------------------------------
> This communication is confidential. We only send and receive email on the
> basis of the terms set out at www.rogers.com/web/content/emailnotice
>
>
>
> Ce message est confidentiel. Notre transmission et réception de courriels
> se fait strictement suivant les modalités énoncées dans l’avis publié à www.rogers.com/aviscourriel
>
> ------------------------------
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://iqug.org/pipermail/iqug/attachments/20191003/e6400004/attachment.html>


More information about the IQUG mailing list