[IQUG] IQUG Digest, Vol 67, Issue 8

Symonds, Jason S jason.s.symonds at jpmorgan.com
Mon Jul 9 02:11:59 MST 2018


Hi,

One thing to note about IQ 16.1 is that SAP upgraded their compiler for IQ16.1 SP03.  From IQ16.1SP03 you need to be on Red Hat 7 as it won't start on Red Hat6.  16.1SP02 works on Red Hat6 still.  See KBA 2551457 for details about 16.1SP03 and Red Hat 7.

Regards,

Jason.

Jason Symonds | Vice President | IB Technology | EQ Database Engineering Team | J.P. Morgan | 5th Floor, 4 John Carpenter Street EC4Y 0JP | T: +44 (0) 20 7134 3890| jason.s.symonds at jpmorgan.com | jpmorgan.com

From: iqug-bounces at iqug.org [mailto:iqug-bounces at iqug.org] On Behalf Of Steve Shen
Sent: 06 July 2018 21:45
To: 'Mumy, Mark'; 'iqug at iqug.org'; 'rwatkins at dssolutions.com'; Baker, Chris
Subject: Re: [IQUG] IQUG Digest, Vol 67, Issue 8

Hi Mark,

Thank you very much for the information. If the client agrees, I will target version 16.1. The important factor is that the IQ version must be compatible with IBM/Cognos versions.

It will be very time consuming to rebuild all the FP default indexes after the IQ instance is migrated to version 16.x.

Thank you again.

Regards,

Steve Shen

-----Original Message-----
From: Mumy, Mark [mailto:mark.mumy at sap.com<mailto:mark.mumy at sap.com>]
Sent: Friday, July 06, 2018 4:40 PM
To: Steve Shen; 'iqug at iqug.org<mailto:iqug at iqug.org>'; 'rwatkins at dssolutions.com<mailto:rwatkins at dssolutions.com>'; Baker, Chris
Subject: Re: IQUG Digest, Vol 67, Issue 8

Don't forget that as part of the migration you will need to rebuild your indexes, especially the default indexes/columns.

Mark

Mark Mumy
Strategic Technology Incubation Group
SAP Global Center of Excellence |  SAP
M +1 347-820-2136 | E mark.mumy at sap.com<mailto:mark.mumy at sap.com>
My Blogs: https://blogs.sap.com/author/markmumy/

https://sap.na.pgiconnect.com/I825063
Conference tel: 18663127353,,8035340905#

On 7/6/18, 3:35 PM, "Mumy, Mark" <mark.mumy at sap.com<mailto:mark.mumy at sap.com>> wrote:

    That's a good process.  The backup/restore allows you to not have to install a version that may or may not work.

    Two ways to go here....  Since you are migrating and going to have to test, you could go with IQ 16.0 or IQ 16.1.  16.1 gives you a longer life as it is the most current release.  IQ 16 is end of life around Nov 2020, so some ways out, but worth keeping in mind.  Also, all new features are going into 16.1.  Something to consider at least.

    Mark

    Mark Mumy
    Strategic Technology Incubation Group
    SAP Global Center of Excellence |  SAP
    M +1 347-820-2136 | E mark.mumy at sap.com<mailto:mark.mumy at sap.com>
    My Blogs: https://blogs.sap.com/author/markmumy/

    https://sap.na.pgiconnect.com/I825063
    Conference tel: 18663127353,,8035340905#

    On 7/6/18, 3:07 PM, "Steve Shen" <sshen at sscinc.com<mailto:sshen at sscinc.com>> wrote:

        Topics:
        1. Cross platform migrations from IQ version 15.4 at Solaris X64 to IQ version 16.0 at Linux (Red Hat version 7.4).
        2. On the most recommended SP at IQ version 16.0 at Linux and Solaris X64

        Hi Mark,

        My existing IQ instances are version 15.4 at Solaris X64. I got the client agreement today to migrate it to 16.0 in the following one or two months.

        The more traditional way for the cross platform migrations will be to migrate IQ version 15.4 to version 16.0 at Solaris X64, take the full backups from IQ version 16.0 at Solaris X64 and load the backups to IQ version 16.0 at Linux. This will eliminate potential version incompatibility issues.

        Please let me know whether the following short-cut will work or not:
        1. I will take the full backups from IQ version 15.4 at Solaris X64.
        2. I will load the full backups from the prior step and load it to version 16.0 at Linux (Red Hat version 7.4).
        3. After the database is loaded to IQ version 16.0 at Linux (Red Hat version 7.4), I will start the IQ version 16 Linux at read-only mode and then issue "alter database upgrade".
        Notes: I prefer to not install IQ version 15.4 to Linux (Red Hat version 7.4) because it may or may not work based on your feedback.

        BTW, which SP at IQ version 16 do you recommend for your clients?

        Your advise will save me time from trail-and-errors.

        Thank you very much for your reminder below.

        Best regards,

        Steve Shen

        -----Original Message-----
        From: Mumy, Mark [mailto:mark.mumy at sap.com<mailto:mark.mumy at sap.com>]
        Sent: Friday, July 06, 2018 2:00 PM
        To: Steve Shen; 'iqug at iqug.org<mailto:iqug at iqug.org>'; 'rwatkins at dssolutions.com<mailto:rwatkins at dssolutions.com>'; Baker, Chris
        Subject: Re: IQUG Digest, Vol 67, Issue 8

        Just to reiterate...  IQ 15 doesn’t support RHEL 7 in any fashion.  It's never been tested, so it may work or it may not.  If it doesn't, well IQ 15 is out of support now, so you'd likely have to drop back to RHEL 6 until you get migrated to IQ 16 and the latest SP and patch.

        Mark

        Mark Mumy
        Strategic Technology Incubation Group
        SAP Global Center of Excellence |  SAP
        M +1 347-820-2136 | E mark.mumy at sap.com<mailto:mark.mumy at sap.com>
        My Blogs: https://blogs.sap.com/author/markmumy/

        https://sap.na.pgiconnect.com/I825063
        Conference tel: 18663127353,,8035340905#

        On 7/6/18, 12:43 PM, "Steve Shen" <sshen at sscinc.com<mailto:sshen at sscinc.com>> wrote:

            Hi all,

            You can ignore the third question because UNIX SAs decided using Red Hat 7.4 with the default file systems "xfs" on SSDs.

            They were not keen in considering Red Hat version 6.x anymore.

            Thank you.

            Regards,

            Steve
            -----Original Message-----
            From: Steve Shen
            Sent: Friday, July 06, 2018 8:59 AM
            To: iqug at iqug.org<mailto:iqug at iqug.org>; 'mark.mumy at sap.com<mailto:mark.mumy at sap.com>'; 'rwatkins at dssolutions.com<mailto:rwatkins at dssolutions.com>'; 'c.baker at sap.com<mailto:c.baker at sap.com>'
            Subject: RE: IQUG Digest, Vol 67, Issue 8

            Today's Topics:

               1. Re: IQUG Digest, Vol 67, Issue 5 (Mumy, Mark)
               2. Re: IQUG Digest, Vol 67, Issue 5 (Steve Shen)
               3. Re: ext3 or ext4 or xfs in Linux (Steve Shen)

            I am adding item #3 above for today's discussion.

            The default file systems at Red Hat version 6.8 and Red Hat version 7.4 were ext4 and xfs respectively in my company.

            Based on my understanding, ext4 is superior to ext3 and xfs is superior to ext4 in general.

            The file system  ext4 allows us to disable journaling to make the file systems perform better.  Is this a common practice to disable journaling in IQ? Will we lose recovery due to turning off journaling?

            Was there a consensus in this IQ community how to determine to use ext3 or ext4 or xfs for any of the following stores?
            1.  IQ Simplex catalog store;
            2. IQ Simpelx data store;
            3. IQ Simplex temp store;
            4. IQ backup file system;

            Notes: We are going to use SnapMirror from the primary IQ machine to the DR host.

            Please let me know if you have the answers.  Thank you.

            Regards,

            Steve Shen

            -----Original Message-----
            From: iqug-bounces at iqug.org<mailto:iqug-bounces at iqug.org> [mailto:iqug-bounces at iqug.org<mailto:iqug-bounces at iqug.org>] On Behalf Of iqug-request at iqug.org<mailto:iqug-request at iqug.org>
            Sent: Thursday, July 05, 2018 12:41 PM
            To: iqug at iqug.org<mailto:iqug at iqug.org>
            Subject: IQUG Digest, Vol 67, Issue 8

            Send IQUG mailing list submissions to
                    iqug at iqug.org<mailto:iqug at iqug.org>

            To subscribe or unsubscribe via the World Wide Web, visit
                    http://iqug.org/mailman/listinfo/iqug
            or, via email, send a message with subject or body 'help' to
                    iqug-request at iqug.org<mailto:iqug-request at iqug.org>

            You can reach the person managing the list at
                    iqug-owner at iqug.org<mailto:iqug-owner at iqug.org>

            When replying, please edit your Subject line so it is more specific than "Re: Contents of IQUG digest..."


            Today's Topics:

               1. Re: IQUG Digest, Vol 67, Issue 5 (Mumy, Mark)
               2. Re: IQUG Digest, Vol 67, Issue 5 (Steve Shen)


            ----------------------------------------------------------------------

            Message: 1
            Date: Thu, 5 Jul 2018 16:26:16 +0000
            From: "Mumy, Mark" <mark.mumy at sap.com<mailto:mark.mumy at sap.com>>
            To: Steve Shen <sshen at sscinc.com<mailto:sshen at sscinc.com>>, "iqug at iqug.org<mailto:iqug at iqug.org>" <iqug at iqug.org<mailto:iqug at iqug.org>>
            Subject: Re: [IQUG] IQUG Digest, Vol 67, Issue 5
            Message-ID: <51C29F4D-3DDD-4A37-B14B-CF9A82D3081B at sap.com<mailto:51C29F4D-3DDD-4A37-B14B-CF9A82D3081B at sap.com>>
            Content-Type: text/plain; charset="utf-8"

            Just saw your questions on RHEL versions.

            See my other email....  Using the manuals from Sybase.com isn't advised as they are not kept up.  For platform specific stuff and OS certifications, you can only get the currently supported platforms from sap.com, here.  What we list in the manuals was the initial versions supported.  We don't update the manuals when we test newer OS versions, we always update the SAP PAM (product availability matrix).
            https://apps.support.sap.com/sap(bD1lbiZjPTAwMQ==)/support/pam/pam.html?smpsrv=https%3a%2f%2fwebsmp106.sap-ag.de#ts=0

            See the IQ 16 manual you point to is grossly out of date and only shows RHEL 5 and 6 as supported when the products were first released.  Use the above link and you will see that other versions are included as well.  That link also includes other platform related details that are important for IQ deployment on various platforms.  Here's the link to all Linux versions supported for IQ 16.0:
            https://apps.support.sap.com/sap/support/pam?hash=s%3D%26filter%3Dfavorites%257CT%26o%3Dmost_viewed%257Cdesc%26st%3Dl%26rpp%3D20%26page%3D1%26pvnr%3D67838200100900005877%26pt%3Dt%257COS%26fclfilter%3DG1%257CLINUX%2520ON%2520X86_64%26ainstnr%3D67838200104900016656

            Our support for OS versions is a tad relaxed.  If it says RHEL 6.2, that doesn?t mean it?s the only version we support.  That's the minimum version we support for the main release of RHEL 6.  Our policy is to support any of the dot releases at that or a higher level.  So not RHEL 6.2, but really RHEL 6.2+ or RHEL 6.x with a minimum of 6.2.  However you want to read it.

            With IQ 15.4, you can use whatever OS you want without fear of violating support.  OK, it's not quite THAT easy.  IQ 15.4 is no longer supported.  So any OS that you want to support would be just fine.

            Do you have an SAP S-number?  The SAP sites contain quite a lot of details on IQ deployments in the PAM, roadmaps, SAP Notes, and manuals.

            Mark

            Mark Mumy
            Strategic Technology Incubation Group
            SAP Global Center of Excellence |  SAP
            M +1 347-820-2136 | E mark.mumy at sap.com<mailto:mark.mumy at sap.com>
            My Blogs: https://blogs.sap.com/author/markmumy/

            https://sap.na.pgiconnect.com/I825063
            Conference tel: 18663127353,,8035340905#

            ?On 7/5/18, 11:03 AM, "iqug-bounces at iqug.org<mailto:iqug-bounces at iqug.org> on behalf of Steve Shen" <iqug-bounces at iqug.org<mailto:iqug-bounces at iqug.org> on behalf of sshen at sscinc.com<mailto:sshen at sscinc.com>> wrote:

                Today's Topics:

                1. Re: What is the IQ performance penalty from using file
                      systems in Linux kernel version 2.6 and above? (Steve Shen)

                2. Re: RedHat Enterprise Linux versions for IQ versions 15.4 and 16.0 (Steve Shen)

                The following are for IQ 15.4 and 16.0 at Linux from the document link http://infocenter.sybase.com/help/topic/com.sybase.infocenter.dc10083.1540/doc/pdf/iqicglin.pdf and http://infocenter.sybase.com/help/topic/com.sybase.infocenter.dc10083.1600/doc/pdf/iqicglin.pdf

                "
                Supported Server Platforms
                Sybase IQ is compatible with these platforms and operating systems.
                * Red Hat Enterprise Linux 6.2 (Linux on POWER; 64-bit)
                * Red Hat Enterprise Linux 6.2 x86-64, Advanced Server and Workstation Editions
                * Red Hat Enterprise Linux 6.0 (Linux onPOWER;64-bit) - minimum patch level Update-2
                * Red Hat Enterprise Linux 6.0 x86-64, Advanced Server and Workstation Editions -
                minimum patch level Update-2
                * Red Hat Enterprise Linux 5.5 (Linux onPOWER;64-bit) - minimum patch level Update-2
                * Red Hat Enterprise Linux 5.5 x86-64, Advanced Server and Workstation Editions -
                minimum patch level Update-2
                * SuSE Linux Enterprise Server (SLES) 11.0 x86-64 - minimum patch level SP1
                * SuSE Linux Enterprise Server (SLES) 11.0 (Linux on POWER; 64-bit) - minimum patch
                level SP1
                Supported Client
                "
                Red Hat Enterprise Linux 5.5 x86-64 or 6.0 x86-64 or 6.2 x86-64 are very old.

                Can I used Red Hat Enterprise Linux version 6.8 or 7.4?  Please advise.  Thank you.

                Regards,

                Steve

                -----Original Message-----
                From: iqug-bounces at iqug.org<mailto:iqug-bounces at iqug.org> [mailto:iqug-bounces at iqug.org<mailto:iqug-bounces at iqug.org>] On Behalf Of iqug-request at iqug.org<mailto:iqug-request at iqug.org>
                Sent: Thursday, July 05, 2018 11:42 AM
                To: iqug at iqug.org<mailto:iqug at iqug.org>
                Subject: IQUG Digest, Vol 67, Issue 5

                Send IQUG mailing list submissions to
                        iqug at iqug.org<mailto:iqug at iqug.org>

                To subscribe or unsubscribe via the World Wide Web, visit
                        http://iqug.org/mailman/listinfo/iqug
                or, via email, send a message with subject or body 'help' to
                        iqug-request at iqug.org<mailto:iqug-request at iqug.org>

                You can reach the person managing the list at
                        iqug-owner at iqug.org<mailto:iqug-owner at iqug.org>

                When replying, please edit your Subject line so it is more specific than "Re: Contents of IQUG digest..."


                Today's Topics:

                   1. Re: What is the IQ performance penalty from using file
                      systems in Linux kernel version 2.6 and above? (Steve Shen)


                ----------------------------------------------------------------------

                Message: 1
                Date: Thu, 5 Jul 2018 15:41:43 +0000
                From: Steve Shen <sshen at sscinc.com<mailto:sshen at sscinc.com>>
                To: "'Mumy, Mark'" <mark.mumy at sap.com<mailto:mark.mumy at sap.com>>, Ron Watkins
                        <rwatkins at dssolutions.com<mailto:rwatkins at dssolutions.com>>, "Baker, Chris" <c.baker at sap.com<mailto:c.baker at sap.com>>,
                        "iqug at iqug.org<mailto:iqug at iqug.org>" <iqug at iqug.org<mailto:iqug at iqug.org>>
                Subject: Re: [IQUG] What is the IQ performance penalty from using file
                        systems in Linux kernel version 2.6 and above?
                Message-ID:
                        <0C03FF7E7FA66E41A61525750FD653399DC52633 at ykt1emxprd6.globeop.com<mailto:0C03FF7E7FA66E41A61525750FD653399DC52633 at ykt1emxprd6.globeop.com>>
                Content-Type: text/plain; charset="utf-8"

                Hi Mark,

                The senior management had chosen ?ease? over ?performance?.

                Thank you very much for your feedback.

                Regards,
                Steve
                From: Mumy, Mark [mailto:mark.mumy at sap.com<mailto:mark.mumy at sap.com>]
                Sent: Thursday, July 05, 2018 11:10 AM
                To: Ron Watkins; Baker, Chris; Steve Shen; iqug at iqug.org<mailto:iqug at iqug.org>
                Subject: Re: [IQUG] What is the IQ performance penalty from using file systems in Linux kernel version 2.6 and above?

                Here?s what it boils down to for me?.

                When you use a filesystem how much control do you have over every aspect of the filesystem, block sizes, continuous or striping methods, SAN communication, SAN/NAS block and stripe sizes, etc???

                The worst answer is, I have no control over any of it.  Best case, you can control some or most of it.

                With raw devices, you have complete control.  Or rather, IQ has complete control over everything outside of how the SAN storage is carved up.

                I had one customer that used filesystems for IQ.  Those nice, big fat IOs that IQ likes to do were cut down to 2k for everything.  Which was then sent back to a SAN that like to do nice big IOs, too.  Another customer had a filesystem that used IOs that were slightly larger than what IQ could use and larger than the SAN block size definition.  It caused 2 IOs for every IO that IQ did because the filesystem write was bigger than a single block on the storage, so it had to be broken into 2 physical IOs.

                Lastly, IQ will now want to open all filesystem devices with O_DIRECT.  So you have to make sure you filesystem supports that as well.  If not, then you leave yourself exposed for a situation where IO could be buffered but not on disk yet.

                There is some merit to using journaling as it can help protect your filesystem.  With IQ, though, it will slow it down.  So you have a choice of performance or filesystem protection.

                In the end, it comes down to ease or performance.  Do you want to easily implement IQ or have complete control over how IQ is implemented so that you can control performance?  There is no right or wrong here.  SAP has chosen that IQ will be implemented on filesystems when it is deployed as part of HANA in dynamic tiering.  That was done because HANA is on filesystem and it is easier to continue that with IQ/DT, though it can be changed if you know enough about HANA and DT to do so.  We made the choice to give up some on performance, for now, so that we could have an easier system to deploy.

                I will say, though, that you had become very good friends with the system and storage administrators.  Chances are something will have to be changed so better to be trusted by them up front so changing something is easier.

                Side note??. I would recommend against using the old Sybase.com man pages.  I know they are much easier to traverse, but they are quite old and not maintained.  This section still holds true, but other items have changed in the newer manuals.

                Mark

                Mark Mumy
                Strategic Technology Incubation Group
                SAP Global Center of Excellence |  SAP
                M +1 347-820-2136 | E mark.mumy at sap.com<mailto:mark.mumy at sap.com><mailto:mark.mumy at sap.com<mailto:mark.mumy at sap.com>>
                My Blogs: https://blogs.sap.com/author/markmumy/

                https://sap.na.pgiconnect.com/I825063
                Conference tel: 18663127353,,8035340905#

                From: "iqug-bounces at iqug.org<mailto:iqug-bounces at iqug.org><mailto:iqug-bounces at iqug.org<mailto:iqug-bounces at iqug.org>>" <iqug-bounces at iqug.org<mailto:iqug-bounces at iqug.org><mailto:iqug-bounces at iqug.org<mailto:iqug-bounces at iqug.org>>> on behalf of Ron Watkins <rwatkins at dssolutions.com<mailto:rwatkins at dssolutions.com><mailto:rwatkins at dssolutions.com<mailto:rwatkins at dssolutions.com>>>
                Date: Thursday, July 5, 2018 at 8:08 AM
                To: Chris Baker <c.baker at sap.com<mailto:c.baker at sap.com><mailto:c.baker at sap.com<mailto:c.baker at sap.com>>>, 'Steve Shen' <sshen at sscinc.com<mailto:sshen at sscinc.com><mailto:sshen at sscinc.com<mailto:sshen at sscinc.com>>>, "iqug at iqug.org<mailto:iqug at iqug.org><mailto:iqug at iqug.org<mailto:iqug at iqug.org>>" <iqug at iqug.org<mailto:iqug at iqug.org><mailto:iqug at iqug.org<mailto:iqug at iqug.org>>>
                Subject: Re: [IQUG] What is the IQ performance penalty from using file systems in Linux kernel version 2.6 and above?

                Journaling is optional and can  be disabled using the following option to mkfs:
                -O ^has_journal
                The other comments I agree with, and always prefer RAW to FS.
                Ron

                From: Baker, Chris [mailto:c.baker at sap.com<mailto:c.baker at sap.com>]
                Sent: Wednesday, July 04, 2018 1:36 PM
                To: Ron Watkins; 'Steve Shen'; iqug at iqug.org<mailto:iqug at iqug.org><mailto:iqug at iqug.org<mailto:iqug at iqug.org>>
                Subject: RE: [IQUG] What is the IQ performance penalty from using file systems in Linux kernel version 2.6 and above?

                To add to Ron?s comments ?

                Direct I/O and write-cache bypassing are good starts, but the filesystem choice WILL affect you as well, even as far as IQ page sizes vs. filesystem page sizes (IQ I/O is broken into smaller pieces).  The Linux I/O elevator can play here as well (anticipatory vs. cfq vs. deadline vs. noop) when defining I/O strategies, even for filesystem.

                How the block device (holding the filesystem) is defined can also affect I/O.  Some SANs allow the blocks to be laid out consecutively, so even if the filesystem pages are smaller that IQ pages, the SAN can ?hold? the write for > 1 page, allowing better performance on the read/write as there is a better chance that consecutive blocks will be part of the same logical page.

                I won?t even mention mirroring and the overhead that may cause.

                Of course it goes without saying that you should not share the IQ filesystem for other purposes, in any case.

                EXT4 is also a journaling filesystem.  It will log every,    single,    access ? and can slow down IO drastically.  You can turn off the EXT4 journaling aspects with various mount options and settings, effectively turning it into EXT3, and this can help performance.

                In the end, you probably want to test RAW vs. Filesystem performance yourself to ensure it is within your own SLAs, performance criteria and I.T. specs.  Don?t overlook what I.T. ?thinks? you need or is only willing to provide, with what you MUST provide for your own SLAs to your end users/customers.

                Chris

                Chris Baker | Database Engineering Evangelist | CERT | PI HANA Platform Data Management | SAP T +1 416-226-7033<tel:+1%20416-226-7033> | M +1 647-224-2033<tel:+1%20647-224-2033> | TF +1 866-716-8860<tel:+1%20866-716-8860>
                SAP Canada Inc. 445 Wes Graham Way, Waterloo, N2L 6R2<x-apple-data-detectors://17/1>
                c.baker at sap.com<mailto:c.baker at sap.com><mailto:c.baker at sap.com<mailto:c.baker at sap.com>> | www.sap.com<http://www.sap.com/>

                https://sap.na.pgiconnect.com/I826572
                Conference tel: 1-866-312-7353,,9648565377#<tel:1-866-312-7353,,9648565377%23>

                From: iqug-bounces at iqug.org<mailto:iqug-bounces at iqug.org><mailto:iqug-bounces at iqug.org<mailto:iqug-bounces at iqug.org>> [mailto:iqug-bounces at iqug.org<mailto:iqug-bounces at iqug.org>] On Behalf Of Ron Watkins
                Sent: Wednesday, July 4, 2018 4:02 PM
                To: 'Steve Shen' <sshen at sscinc.com<mailto:sshen at sscinc.com><mailto:sshen at sscinc.com<mailto:sshen at sscinc.com>>>; iqug at iqug.org<mailto:iqug at iqug.org><mailto:iqug at iqug.org<mailto:iqug at iqug.org>>
                Subject: Re: [IQUG] What is the IQ performance penalty from using file systems in Linux kernel version 2.6 and above?

                You can continue to use RAW devices under RHEL, and RHEL 7.4 still supports ext4 filesystems also.
                Performance is generally dictated by the storage itself, and is less dependent on the ?format? of the storage.
                Thus, if you have multiple disks providing multiple paths to storage you will see a greater improvement over a single disk.
                RAW devices do perform better than filesystem, however, it?s not possible to put an exact value on how much better as they are both strongly dependent on the underlying storage.
                Ron

                From: iqug-bounces at iqug.org<mailto:iqug-bounces at iqug.org><mailto:iqug-bounces at iqug.org<mailto:iqug-bounces at iqug.org>> [mailto:iqug-bounces at iqug.org<mailto:iqug-bounces at iqug.org>] On Behalf Of Steve Shen
                Sent: Wednesday, July 04, 2018 12:15 PM
                To: iqug at iqug.org<mailto:iqug at iqug.org><mailto:iqug at iqug.org<mailto:iqug at iqug.org>>
                Subject: [IQUG] What is the IQ performance penalty from using file systems in Linux kernel version 2.6 and above?

                Today?s Topic:

                (1). What is the IQ performance penalty from using file systems relative to using true raw partitions for main store and temporary store in Linux kernel version 2.6 and above?

                We are considering to migrate our IQ Simplex from Solaris X64 to RedHat version 6.8 or 7.4. They are Linux kernel version 2.6 and above.

                RedHat version 6.8 uses ?ext4? file system.  RedHat version 7.4 used ?xfs? file system.

                This discussion is for IQ Simplex only.  I am very aware that using file systems will prevent me from converting the Simplex to Multiplex later.

                Based on the documentation below, it seemed okay in using file systems at Linux kernel version 2.6 and above:

                http://infocenter.sybase.com/help/topic/com.sybase.infocenter.dc00169.1600/doc/pdf/iqperf.pdf
                ?
                Controlling File System Buffering
                On some file systems, you can turn file system buffering on or off. Turning file system buffering off usually reduces paging and improves performance.
                To disable file system buffering for IQ Main dbspaces of existing databases, issue the following statement:
                SET OPTION "PUBLIC".OS_FILE_CACHE_BUFFERING = OFF To disable file system buffering for IQ Temporary dbspaces of existing databases, issue the following statement:
                SET OPTION "PUBLIC".OS_FILE_CACHE_BUFFERING_TEMPDB = OFF You can only set this option for the PUBLIC role. Shut down the database and restart it for the change to take effect.
                This direct I/O performance option is available on Solaris UFS, Linux, Linux IBM, AIX, and Windows file systems only. This option has no effect on HP-UX and HP-UXi and does not affect databases on raw disk. In Linux, direct I/O is supported in kernel versions 2.6.x To enable direct I/O on Linux kernel version 2.6 and AIX, also set the environment variable IQ_USE_DIRECTIO to 1. Direct I/O is disabled by default in Linux kernel version 2.6 and AIX. IQ_USE_DIRECTIO has no effect on Solaris and Windows.
                ?

                It?s also documented by Mark that using file systems is normally slower than true raw partitions in Linux:

                https://archive.sap.com/discussions/thread/3518966

                Did any of us ever do apple-to-apple comparisons and measure the performance penalty relative to using true raw partitions?

                Was it 5%, 10%, 15% or 20% performance penalty or more from using file systems instead of true raw partitions at Linux?

                Thank you.

                Happy Independency Day!
                Steve Shen
                SS&C Technologies

                This email with all information contained herein or attached hereto may contain confidential and/or privileged information intended for the addressee(s) only. If you have received this email in error, please contact the sender and immediately delete this email in its entirety and any attachments thereto.
                This email with all information contained herein or attached hereto may contain confidential and/or privileged information intended for the addressee(s) only. If you have received this email in error, please contact the sender and immediately delete this email in its entirety and any attachments thereto.
                -------------- next part --------------
                An HTML attachment was scrubbed...
                URL: <http://iqug.org/pipermail/iqug/attachments/20180705/acb40d61/attachment.html>

                ------------------------------

                _______________________________________________
                IQUG mailing list
                IQUG at iqug.org<mailto:IQUG at iqug.org>
                http://iqug.org/mailman/listinfo/iqug

                End of IQUG Digest, Vol 67, Issue 5
                ***********************************
                This email with all information contained herein or attached hereto may contain confidential and/or privileged information intended for the addressee(s) only. If you have received this email in error, please contact the sender and immediately delete this email in its entirety and any attachments thereto.
                _______________________________________________
                IQUG mailing list
                IQUG at iqug.org<mailto:IQUG at iqug.org>
                http://iqug.org/mailman/listinfo/iqug



            ------------------------------

            Message: 2
            Date: Thu, 5 Jul 2018 16:40:27 +0000
            From: Steve Shen <sshen at sscinc.com<mailto:sshen at sscinc.com>>
            To: "'Mumy, Mark'" <mark.mumy at sap.com<mailto:mark.mumy at sap.com>>, "iqug at iqug.org<mailto:iqug at iqug.org>"
                    <iqug at iqug.org<mailto:iqug at iqug.org>>
            Subject: Re: [IQUG] IQUG Digest, Vol 67, Issue 5
            Message-ID:
                    <0C03FF7E7FA66E41A61525750FD653399DC5397C at ykt1emxprd6.globeop.com<mailto:0C03FF7E7FA66E41A61525750FD653399DC5397C at ykt1emxprd6.globeop.com>>
            Content-Type: text/plain; charset="utf-8"

            Hi Mark,

            I should not go back to Sybase web-sites anymore.  I will go to SAP web-site from now on.

            Thank you for your reminder.

            Regards,

            Steve

            -----Original Message-----
            From: Mumy, Mark [mailto:mark.mumy at sap.com<mailto:mark.mumy at sap.com>]
            Sent: Thursday, July 05, 2018 12:26 PM
            To: Steve Shen; iqug at iqug.org<mailto:iqug at iqug.org>
            Subject: Re: [IQUG] IQUG Digest, Vol 67, Issue 5

            Just saw your questions on RHEL versions.

            See my other email....  Using the manuals from Sybase.com isn't advised as they are not kept up.  For platform specific stuff and OS certifications, you can only get the currently supported platforms from sap.com, here.  What we list in the manuals was the initial versions supported.  We don't update the manuals when we test newer OS versions, we always update the SAP PAM (product availability matrix).
            https://apps.support.sap.com/sap(bD1lbiZjPTAwMQ==)/support/pam/pam.html?smpsrv=https%3a%2f%2fwebsmp106.sap-ag.de#ts=0

            See the IQ 16 manual you point to is grossly out of date and only shows RHEL 5 and 6 as supported when the products were first released.  Use the above link and you will see that other versions are included as well.  That link also includes other platform related details that are important for IQ deployment on various platforms.  Here's the link to all Linux versions supported for IQ 16.0:
            https://apps.support.sap.com/sap/support/pam?hash=s%3D%26filter%3Dfavorites%257CT%26o%3Dmost_viewed%257Cdesc%26st%3Dl%26rpp%3D20%26page%3D1%26pvnr%3D67838200100900005877%26pt%3Dt%257COS%26fclfilter%3DG1%257CLINUX%2520ON%2520X86_64%26ainstnr%3D67838200104900016656

            Our support for OS versions is a tad relaxed.  If it says RHEL 6.2, that doesn?t mean it?s the only version we support.  That's the minimum version we support for the main release of RHEL 6.  Our policy is to support any of the dot releases at that or a higher level.  So not RHEL 6.2, but really RHEL 6.2+ or RHEL 6.x with a minimum of 6.2.  However you want to read it.

            With IQ 15.4, you can use whatever OS you want without fear of violating support.  OK, it's not quite THAT easy.  IQ 15.4 is no longer supported.  So any OS that you want to support would be just fine.

            Do you have an SAP S-number?  The SAP sites contain quite a lot of details on IQ deployments in the PAM, roadmaps, SAP Notes, and manuals.

            Mark

            Mark Mumy
            Strategic Technology Incubation Group
            SAP Global Center of Excellence |  SAP
            M +1 347-820-2136 | E mark.mumy at sap.com<mailto:mark.mumy at sap.com>
            My Blogs: https://blogs.sap.com/author/markmumy/

            https://sap.na.pgiconnect.com/I825063
            Conference tel: 18663127353,,8035340905#

            ?On 7/5/18, 11:03 AM, "iqug-bounces at iqug.org<mailto:iqug-bounces at iqug.org> on behalf of Steve Shen" <iqug-bounces at iqug.org<mailto:iqug-bounces at iqug.org> on behalf of sshen at sscinc.com<mailto:sshen at sscinc.com>> wrote:

                Today's Topics:

                1. Re: What is the IQ performance penalty from using file
                      systems in Linux kernel version 2.6 and above? (Steve Shen)

                2. Re: RedHat Enterprise Linux versions for IQ versions 15.4 and 16.0 (Steve Shen)

                The following are for IQ 15.4 and 16.0 at Linux from the document link http://infocenter.sybase.com/help/topic/com.sybase.infocenter.dc10083.1540/doc/pdf/iqicglin.pdf and http://infocenter.sybase.com/help/topic/com.sybase.infocenter.dc10083.1600/doc/pdf/iqicglin.pdf

                "
                Supported Server Platforms
                Sybase IQ is compatible with these platforms and operating systems.
                * Red Hat Enterprise Linux 6.2 (Linux on POWER; 64-bit)
                * Red Hat Enterprise Linux 6.2 x86-64, Advanced Server and Workstation Editions
                * Red Hat Enterprise Linux 6.0 (Linux onPOWER;64-bit) - minimum patch level Update-2
                * Red Hat Enterprise Linux 6.0 x86-64, Advanced Server and Workstation Editions -
                minimum patch level Update-2
                * Red Hat Enterprise Linux 5.5 (Linux onPOWER;64-bit) - minimum patch level Update-2
                * Red Hat Enterprise Linux 5.5 x86-64, Advanced Server and Workstation Editions -
                minimum patch level Update-2
                * SuSE Linux Enterprise Server (SLES) 11.0 x86-64 - minimum patch level SP1
                * SuSE Linux Enterprise Server (SLES) 11.0 (Linux on POWER; 64-bit) - minimum patch
                level SP1
                Supported Client
                "
                Red Hat Enterprise Linux 5.5 x86-64 or 6.0 x86-64 or 6.2 x86-64 are very old.

                Can I used Red Hat Enterprise Linux version 6.8 or 7.4?  Please advise.  Thank you.

                Regards,

                Steve

                -----Original Message-----
                From: iqug-bounces at iqug.org<mailto:iqug-bounces at iqug.org> [mailto:iqug-bounces at iqug.org<mailto:iqug-bounces at iqug.org>] On Behalf Of iqug-request at iqug.org<mailto:iqug-request at iqug.org>
                Sent: Thursday, July 05, 2018 11:42 AM
                To: iqug at iqug.org<mailto:iqug at iqug.org>
                Subject: IQUG Digest, Vol 67, Issue 5

                Send IQUG mailing list submissions to
                        iqug at iqug.org<mailto:iqug at iqug.org>

                To subscribe or unsubscribe via the World Wide Web, visit
                        http://iqug.org/mailman/listinfo/iqug
                or, via email, send a message with subject or body 'help' to
                        iqug-request at iqug.org<mailto:iqug-request at iqug.org>

                You can reach the person managing the list at
                        iqug-owner at iqug.org<mailto:iqug-owner at iqug.org>

                When replying, please edit your Subject line so it is more specific than "Re: Contents of IQUG digest..."


                Today's Topics:

                   1. Re: What is the IQ performance penalty from using file
                      systems in Linux kernel version 2.6 and above? (Steve Shen)


                ----------------------------------------------------------------------

                Message: 1
                Date: Thu, 5 Jul 2018 15:41:43 +0000
                From: Steve Shen <sshen at sscinc.com<mailto:sshen at sscinc.com>>
                To: "'Mumy, Mark'" <mark.mumy at sap.com<mailto:mark.mumy at sap.com>>, Ron Watkins
                        <rwatkins at dssolutions.com<mailto:rwatkins at dssolutions.com>>, "Baker, Chris" <c.baker at sap.com<mailto:c.baker at sap.com>>,
                        "iqug at iqug.org<mailto:iqug at iqug.org>" <iqug at iqug.org<mailto:iqug at iqug.org>>
                Subject: Re: [IQUG] What is the IQ performance penalty from using file
                        systems in Linux kernel version 2.6 and above?
                Message-ID:
                        <0C03FF7E7FA66E41A61525750FD653399DC52633 at ykt1emxprd6.globeop.com<mailto:0C03FF7E7FA66E41A61525750FD653399DC52633 at ykt1emxprd6.globeop.com>>
                Content-Type: text/plain; charset="utf-8"

                Hi Mark,

                The senior management had chosen ?ease? over ?performance?.

                Thank you very much for your feedback.

                Regards,
                Steve
                From: Mumy, Mark [mailto:mark.mumy at sap.com<mailto:mark.mumy at sap.com>]
                Sent: Thursday, July 05, 2018 11:10 AM
                To: Ron Watkins; Baker, Chris; Steve Shen; iqug at iqug.org<mailto:iqug at iqug.org>
                Subject: Re: [IQUG] What is the IQ performance penalty from using file systems in Linux kernel version 2.6 and above?

                Here?s what it boils down to for me?.

                When you use a filesystem how much control do you have over every aspect of the filesystem, block sizes, continuous or striping methods, SAN communication, SAN/NAS block and stripe sizes, etc???

                The worst answer is, I have no control over any of it.  Best case, you can control some or most of it.

                With raw devices, you have complete control.  Or rather, IQ has complete control over everything outside of how the SAN storage is carved up.

                I had one customer that used filesystems for IQ.  Those nice, big fat IOs that IQ likes to do were cut down to 2k for everything.  Which was then sent back to a SAN that like to do nice big IOs, too.  Another customer had a filesystem that used IOs that were slightly larger than what IQ could use and larger than the SAN block size definition.  It caused 2 IOs for every IO that IQ did because the filesystem write was bigger than a single block on the storage, so it had to be broken into 2 physical IOs.

                Lastly, IQ will now want to open all filesystem devices with O_DIRECT.  So you have to make sure you filesystem supports that as well.  If not, then you leave yourself exposed for a situation where IO could be buffered but not on disk yet.

                There is some merit to using journaling as it can help protect your filesystem.  With IQ, though, it will slow it down.  So you have a choice of performance or filesystem protection.

                In the end, it comes down to ease or performance.  Do you want to easily implement IQ or have complete control over how IQ is implemented so that you can control performance?  There is no right or wrong here.  SAP has chosen that IQ will be implemented on filesystems when it is deployed as part of HANA in dynamic tiering.  That was done because HANA is on filesystem and it is easier to continue that with IQ/DT, though it can be changed if you know enough about HANA and DT to do so.  We made the choice to give up some on performance, for now, so that we could have an easier system to deploy.

                I will say, though, that you had become very good friends with the system and storage administrators.  Chances are something will have to be changed so better to be trusted by them up front so changing something is easier.

                Side note??. I would recommend against using the old Sybase.com man pages.  I know they are much easier to traverse, but they are quite old and not maintained.  This section still holds true, but other items have changed in the newer manuals.

                Mark

                Mark Mumy
                Strategic Technology Incubation Group
                SAP Global Center of Excellence |  SAP
                M +1 347-820-2136 | E mark.mumy at sap.com<mailto:mark.mumy at sap.com><mailto:mark.mumy at sap.com<mailto:mark.mumy at sap.com>>
                My Blogs: https://blogs.sap.com/author/markmumy/

                https://sap.na.pgiconnect.com/I825063
                Conference tel: 18663127353,,8035340905#

                From: "iqug-bounces at iqug.org<mailto:iqug-bounces at iqug.org><mailto:iqug-bounces at iqug.org<mailto:iqug-bounces at iqug.org>>" <iqug-bounces at iqug.org<mailto:iqug-bounces at iqug.org><mailto:iqug-bounces at iqug.org<mailto:iqug-bounces at iqug.org>>> on behalf of Ron Watkins <rwatkins at dssolutions.com<mailto:rwatkins at dssolutions.com><mailto:rwatkins at dssolutions.com<mailto:rwatkins at dssolutions.com>>>
                Date: Thursday, July 5, 2018 at 8:08 AM
                To: Chris Baker <c.baker at sap.com<mailto:c.baker at sap.com><mailto:c.baker at sap.com<mailto:c.baker at sap.com>>>, 'Steve Shen' <sshen at sscinc.com<mailto:sshen at sscinc.com><mailto:sshen at sscinc.com<mailto:sshen at sscinc.com>>>, "iqug at iqug.org<mailto:iqug at iqug.org><mailto:iqug at iqug.org<mailto:iqug at iqug.org>>" <iqug at iqug.org<mailto:iqug at iqug.org><mailto:iqug at iqug.org<mailto:iqug at iqug.org>>>
                Subject: Re: [IQUG] What is the IQ performance penalty from using file systems in Linux kernel version 2.6 and above?

                Journaling is optional and can  be disabled using the following option to mkfs:
                -O ^has_journal
                The other comments I agree with, and always prefer RAW to FS.
                Ron

                From: Baker, Chris [mailto:c.baker at sap.com<mailto:c.baker at sap.com>]
                Sent: Wednesday, July 04, 2018 1:36 PM
                To: Ron Watkins; 'Steve Shen'; iqug at iqug.org<mailto:iqug at iqug.org><mailto:iqug at iqug.org<mailto:iqug at iqug.org>>
                Subject: RE: [IQUG] What is the IQ performance penalty from using file systems in Linux kernel version 2.6 and above?

                To add to Ron?s comments ?

                Direct I/O and write-cache bypassing are good starts, but the filesystem choice WILL affect you as well, even as far as IQ page sizes vs. filesystem page sizes (IQ I/O is broken into smaller pieces).  The Linux I/O elevator can play here as well (anticipatory vs. cfq vs. deadline vs. noop) when defining I/O strategies, even for filesystem.

                How the block device (holding the filesystem) is defined can also affect I/O.  Some SANs allow the blocks to be laid out consecutively, so even if the filesystem pages are smaller that IQ pages, the SAN can ?hold? the write for > 1 page, allowing better performance on the read/write as there is a better chance that consecutive blocks will be part of the same logical page.

                I won?t even mention mirroring and the overhead that may cause.

                Of course it goes without saying that you should not share the IQ filesystem for other purposes, in any case.

                EXT4 is also a journaling filesystem.  It will log every,    single,    access ? and can slow down IO drastically.  You can turn off the EXT4 journaling aspects with various mount options and settings, effectively turning it into EXT3, and this can help performance.

                In the end, you probably want to test RAW vs. Filesystem performance yourself to ensure it is within your own SLAs, performance criteria and I.T. specs.  Don?t overlook what I.T. ?thinks? you need or is only willing to provide, with what you MUST provide for your own SLAs to your end users/customers.

                Chris

                Chris Baker | Database Engineering Evangelist | CERT | PI HANA Platform Data Management | SAP T +1 416-226-7033<tel:+1%20416-226-7033> | M +1 647-224-2033<tel:+1%20647-224-2033> | TF +1 866-716-8860<tel:+1%20866-716-8860>
                SAP Canada Inc. 445 Wes Graham Way, Waterloo, N2L 6R2<x-apple-data-detectors://17/1>
                c.baker at sap.com<mailto:c.baker at sap.com><mailto:c.baker at sap.com<mailto:c.baker at sap.com>> | www.sap.com<http://www.sap.com/>

                https://sap.na.pgiconnect.com/I826572
                Conference tel: 1-866-312-7353,,9648565377#<tel:1-866-312-7353,,9648565377%23>

                From: iqug-bounces at iqug.org<mailto:iqug-bounces at iqug.org><mailto:iqug-bounces at iqug.org<mailto:iqug-bounces at iqug.org>> [mailto:iqug-bounces at iqug.org<mailto:iqug-bounces at iqug.org>] On Behalf Of Ron Watkins
                Sent: Wednesday, July 4, 2018 4:02 PM
                To: 'Steve Shen' <sshen at sscinc.com<mailto:sshen at sscinc.com><mailto:sshen at sscinc.com<mailto:sshen at sscinc.com>>>; iqug at iqug.org<mailto:iqug at iqug.org><mailto:iqug at iqug.org<mailto:iqug at iqug.org>>
                Subject: Re: [IQUG] What is the IQ performance penalty from using file systems in Linux kernel version 2.6 and above?

                You can continue to use RAW devices under RHEL, and RHEL 7.4 still supports ext4 filesystems also.
                Performance is generally dictated by the storage itself, and is less dependent on the ?format? of the storage.
                Thus, if you have multiple disks providing multiple paths to storage you will see a greater improvement over a single disk.
                RAW devices do perform better than filesystem, however, it?s not possible to put an exact value on how much better as they are both strongly dependent on the underlying storage.
                Ron

                From: iqug-bounces at iqug.org<mailto:iqug-bounces at iqug.org><mailto:iqug-bounces at iqug.org<mailto:iqug-bounces at iqug.org>> [mailto:iqug-bounces at iqug.org<mailto:iqug-bounces at iqug.org>] On Behalf Of Steve Shen
                Sent: Wednesday, July 04, 2018 12:15 PM
                To: iqug at iqug.org<mailto:iqug at iqug.org><mailto:iqug at iqug.org<mailto:iqug at iqug.org>>
                Subject: [IQUG] What is the IQ performance penalty from using file systems in Linux kernel version 2.6 and above?

                Today?s Topic:

                (1). What is the IQ performance penalty from using file systems relative to using true raw partitions for main store and temporary store in Linux kernel version 2.6 and above?

                We are considering to migrate our IQ Simplex from Solaris X64 to RedHat version 6.8 or 7.4. They are Linux kernel version 2.6 and above.

                RedHat version 6.8 uses ?ext4? file system.  RedHat version 7.4 used ?xfs? file system.

                This discussion is for IQ Simplex only.  I am very aware that using file systems will prevent me from converting the Simplex to Multiplex later.

                Based on the documentation below, it seemed okay in using file systems at Linux kernel version 2.6 and above:

                http://infocenter.sybase.com/help/topic/com.sybase.infocenter.dc00169.1600/doc/pdf/iqperf.pdf
                ?
                Controlling File System Buffering
                On some file systems, you can turn file system buffering on or off. Turning file system buffering off usually reduces paging and improves performance.
                To disable file system buffering for IQ Main dbspaces of existing databases, issue the following statement:
                SET OPTION "PUBLIC".OS_FILE_CACHE_BUFFERING = OFF To disable file system buffering for IQ Temporary dbspaces of existing databases, issue the following statement:
                SET OPTION "PUBLIC".OS_FILE_CACHE_BUFFERING_TEMPDB = OFF You can only set this option for the PUBLIC role. Shut down the database and restart it for the change to take effect.
                This direct I/O performance option is available on Solaris UFS, Linux, Linux IBM, AIX, and Windows file systems only. This option has no effect on HP-UX and HP-UXi and does not affect databases on raw disk. In Linux, direct I/O is supported in kernel versions 2.6.x To enable direct I/O on Linux kernel version 2.6 and AIX, also set the environment variable IQ_USE_DIRECTIO to 1. Direct I/O is disabled by default in Linux kernel version 2.6 and AIX. IQ_USE_DIRECTIO has no effect on Solaris and Windows.
                ?

                It?s also documented by Mark that using file systems is normally slower than true raw partitions in Linux:

                https://archive.sap.com/discussions/thread/3518966

                Did any of us ever do apple-to-apple comparisons and measure the performance penalty relative to using true raw partitions?

                Was it 5%, 10%, 15% or 20% performance penalty or more from using file systems instead of true raw partitions at Linux?

                Thank you.

                Happy Independency Day!
                Steve Shen
                SS&C Technologies

                This email with all information contained herein or attached hereto may contain confidential and/or privileged information intended for the addressee(s) only. If you have received this email in error, please contact the sender and immediately delete this email in its entirety and any attachments thereto.
                This email with all information contained herein or attached hereto may contain confidential and/or privileged information intended for the addressee(s) only. If you have received this email in error, please contact the sender and immediately delete this email in its entirety and any attachments thereto.
                -------------- next part --------------
                An HTML attachment was scrubbed...
                URL: <http://iqug.org/pipermail/iqug/attachments/20180705/acb40d61/attachment.html>

                ------------------------------

                _______________________________________________
                IQUG mailing list
                IQUG at iqug.org<mailto:IQUG at iqug.org>
                http://iqug.org/mailman/listinfo/iqug

                End of IQUG Digest, Vol 67, Issue 5
                ***********************************
                This email with all information contained herein or attached hereto may contain confidential and/or privileged information intended for the addressee(s) only. If you have received this email in error, please contact the sender and immediately delete this email in its entirety and any attachments thereto.
                _______________________________________________
                IQUG mailing list
                IQUG at iqug.org<mailto:IQUG at iqug.org>
                http://iqug.org/mailman/listinfo/iqug


            This email with all information contained herein or attached hereto may contain confidential and/or privileged information intended for the addressee(s) only. If you have received this email in error, please contact the sender and immediately delete this email in its entirety and any attachments thereto.

            ------------------------------

            _______________________________________________
            IQUG mailing list
            IQUG at iqug.org<mailto:IQUG at iqug.org>
            http://iqug.org/mailman/listinfo/iqug

            End of IQUG Digest, Vol 67, Issue 8
            ***********************************
            This email with all information contained herein or attached hereto may contain confidential and/or privileged information intended for the addressee(s) only. If you have received this email in error, please contact the sender and immediately delete this email in its entirety and any attachments thereto.


        This email with all information contained herein or attached hereto may contain confidential and/or privileged information intended for the addressee(s) only. If you have received this email in error, please contact the sender and immediately delete this email in its entirety and any attachments thereto.




This email with all information contained herein or attached hereto may contain confidential and/or privileged information intended for the addressee(s) only. If you have received this email in error, please contact the sender and immediately delete this email in its entirety and any attachments thereto.
_______________________________________________
IQUG mailing list
IQUG at iqug.org<mailto:IQUG at iqug.org>
http://iqug.org/mailman/listinfo/iqug

This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://iqug.org/pipermail/iqug/attachments/20180709/9d1ce214/attachment-0001.html>


More information about the IQUG mailing list