[IQUG] What is the IQ performance penalty from using file systems in Linux kernel version 2.6 and above?

Mumy, Mark mark.mumy at sap.com
Thu Jul 5 08:09:46 MST 2018


Here’s what it boils down to for me….

When you use a filesystem how much control do you have over every aspect of the filesystem, block sizes, continuous or striping methods, SAN communication, SAN/NAS block and stripe sizes, etc???

The worst answer is, I have no control over any of it.  Best case, you can control some or most of it.

With raw devices, you have complete control.  Or rather, IQ has complete control over everything outside of how the SAN storage is carved up.

I had one customer that used filesystems for IQ.  Those nice, big fat IOs that IQ likes to do were cut down to 2k for everything.  Which was then sent back to a SAN that like to do nice big IOs, too.  Another customer had a filesystem that used IOs that were slightly larger than what IQ could use and larger than the SAN block size definition.  It caused 2 IOs for every IO that IQ did because the filesystem write was bigger than a single block on the storage, so it had to be broken into 2 physical IOs.

Lastly, IQ will now want to open all filesystem devices with O_DIRECT.  So you have to make sure you filesystem supports that as well.  If not, then you leave yourself exposed for a situation where IO could be buffered but not on disk yet.

There is some merit to using journaling as it can help protect your filesystem.  With IQ, though, it will slow it down.  So you have a choice of performance or filesystem protection.

In the end, it comes down to ease or performance.  Do you want to easily implement IQ or have complete control over how IQ is implemented so that you can control performance?  There is no right or wrong here.  SAP has chosen that IQ will be implemented on filesystems when it is deployed as part of HANA in dynamic tiering.  That was done because HANA is on filesystem and it is easier to continue that with IQ/DT, though it can be changed if you know enough about HANA and DT to do so.  We made the choice to give up some on performance, for now, so that we could have an easier system to deploy.

I will say, though, that you had become very good friends with the system and storage administrators.  Chances are something will have to be changed so better to be trusted by them up front so changing something is easier.

Side note……. I would recommend against using the old Sybase.com man pages.  I know they are much easier to traverse, but they are quite old and not maintained.  This section still holds true, but other items have changed in the newer manuals.

Mark

Mark Mumy
Strategic Technology Incubation Group
SAP Global Center of Excellence |  SAP
M +1 347-820-2136 | E mark.mumy at sap.com<mailto:mark.mumy at sap.com>
My Blogs: https://blogs.sap.com/author/markmumy/

https://sap.na.pgiconnect.com/I825063
Conference tel: 18663127353,,8035340905#

From: "iqug-bounces at iqug.org" <iqug-bounces at iqug.org> on behalf of Ron Watkins <rwatkins at dssolutions.com>
Date: Thursday, July 5, 2018 at 8:08 AM
To: Chris Baker <c.baker at sap.com>, 'Steve Shen' <sshen at sscinc.com>, "iqug at iqug.org" <iqug at iqug.org>
Subject: Re: [IQUG] What is the IQ performance penalty from using file systems in Linux kernel version 2.6 and above?

Journaling is optional and can  be disabled using the following option to mkfs:
-O ^has_journal
The other comments I agree with, and always prefer RAW to FS.
Ron

From: Baker, Chris [mailto:c.baker at sap.com]
Sent: Wednesday, July 04, 2018 1:36 PM
To: Ron Watkins; 'Steve Shen'; iqug at iqug.org
Subject: RE: [IQUG] What is the IQ performance penalty from using file systems in Linux kernel version 2.6 and above?

To add to Ron’s comments –

Direct I/O and write-cache bypassing are good starts, but the filesystem choice WILL affect you as well, even as far as IQ page sizes vs. filesystem page sizes (IQ I/O is broken into smaller pieces).  The Linux I/O elevator can play here as well (anticipatory vs. cfq vs. deadline vs. noop) when defining I/O strategies, even for filesystem.

How the block device (holding the filesystem) is defined can also affect I/O.  Some SANs allow the blocks to be laid out consecutively, so even if the filesystem pages are smaller that IQ pages, the SAN can ‘hold’ the write for > 1 page, allowing better performance on the read/write as there is a better chance that consecutive blocks will be part of the same logical page.

I won’t even mention mirroring and the overhead that may cause.

Of course it goes without saying that you should not share the IQ filesystem for other purposes, in any case.

EXT4 is also a journaling filesystem.  It will log every,    single,    access – and can slow down IO drastically.  You can turn off the EXT4 journaling aspects with various mount options and settings, effectively turning it into EXT3, and this can help performance.

In the end, you probably want to test RAW vs. Filesystem performance yourself to ensure it is within your own SLAs, performance criteria and I.T. specs.  Don’t overlook what I.T. ‘thinks’ you need or is only willing to provide, with what you MUST provide for your own SLAs to your end users/customers.

Chris

Chris Baker | Database Engineering Evangelist | CERT | PI HANA Platform Data Management | SAP
T +1 416-226-7033<tel:+1%20416-226-7033> | M +1 647-224-2033<tel:+1%20647-224-2033> | TF +1 866-716-8860<tel:+1%20866-716-8860>
SAP Canada Inc. 445 Wes Graham Way, Waterloo, N2L 6R2<x-apple-data-detectors://17/1>
c.baker at sap.com<mailto:c.baker at sap.com> | www.sap.com<http://www.sap.com/>

https://sap.na.pgiconnect.com/I826572
Conference tel: 1-866-312-7353,,9648565377#<tel:1-866-312-7353,,9648565377%23>

From: iqug-bounces at iqug.org [mailto:iqug-bounces at iqug.org] On Behalf Of Ron Watkins
Sent: Wednesday, July 4, 2018 4:02 PM
To: 'Steve Shen' <sshen at sscinc.com>; iqug at iqug.org
Subject: Re: [IQUG] What is the IQ performance penalty from using file systems in Linux kernel version 2.6 and above?

You can continue to use RAW devices under RHEL, and RHEL 7.4 still supports ext4 filesystems also.
Performance is generally dictated by the storage itself, and is less dependent on the “format” of the storage.
Thus, if you have multiple disks providing multiple paths to storage you will see a greater improvement over a single disk.
RAW devices do perform better than filesystem, however, it’s not possible to put an exact value on how much better as they are both strongly dependent on the underlying storage.
Ron

From: iqug-bounces at iqug.org<mailto:iqug-bounces at iqug.org> [mailto:iqug-bounces at iqug.org] On Behalf Of Steve Shen
Sent: Wednesday, July 04, 2018 12:15 PM
To: iqug at iqug.org<mailto:iqug at iqug.org>
Subject: [IQUG] What is the IQ performance penalty from using file systems in Linux kernel version 2.6 and above?

Today’s Topic:

(1). What is the IQ performance penalty from using file systems relative to using true raw partitions for main store and temporary store in Linux kernel version 2.6 and above?

We are considering to migrate our IQ Simplex from Solaris X64 to RedHat version 6.8 or 7.4. They are Linux kernel version 2.6 and above.

RedHat version 6.8 uses “ext4” file system.  RedHat version 7.4 used “xfs” file system.

This discussion is for IQ Simplex only.  I am very aware that using file systems will prevent me from converting the Simplex to Multiplex later.

Based on the documentation below, it seemed okay in using file systems at Linux kernel version 2.6 and above:

http://infocenter.sybase.com/help/topic/com.sybase.infocenter.dc00169.1600/doc/pdf/iqperf.pdf
“
Controlling File System Buffering
On some file systems, you can turn file system buffering on or off. Turning file system
buffering off usually reduces paging and improves performance.
To disable file system buffering for IQ Main dbspaces of existing databases, issue the
following statement:
SET OPTION "PUBLIC".OS_FILE_CACHE_BUFFERING = OFF
To disable file system buffering for IQ Temporary dbspaces of existing databases, issue the
following statement:
SET OPTION "PUBLIC".OS_FILE_CACHE_BUFFERING_TEMPDB = OFF
You can only set this option for the PUBLIC role. Shut down the database and restart it for the
change to take effect.
This direct I/O performance option is available on Solaris UFS, Linux, Linux IBM, AIX, and
Windows file systems only. This option has no effect on HP-UX and HP-UXi and does not
affect databases on raw disk. In Linux, direct I/O is supported in kernel versions 2.6.x
To enable direct I/O on Linux kernel version 2.6 and AIX, also set the environment variable
IQ_USE_DIRECTIO to 1. Direct I/O is disabled by default in Linux kernel version 2.6 and
AIX. IQ_USE_DIRECTIO has no effect on Solaris and Windows.
“

It’s also documented by Mark that using file systems is normally slower than true raw partitions in Linux:

https://archive.sap.com/discussions/thread/3518966

Did any of us ever do apple-to-apple comparisons and measure the performance penalty relative to using true raw partitions?

Was it 5%, 10%, 15% or 20% performance penalty or more from using file systems instead of true raw partitions at Linux?

Thank you.

Happy Independency Day!
Steve Shen
SS&C Technologies

This email with all information contained herein or attached hereto may contain confidential and/or privileged information intended for the addressee(s) only. If you have received this email in error, please contact the sender and immediately delete this email in its entirety and any attachments thereto.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://iqug.org/pipermail/iqug/attachments/20180705/d83006d8/attachment-0001.html>


More information about the IQUG mailing list