[e2eTESTING] V tests: Kerchunk vs Pyfive engines

[valeriupredoi]

Local tests on V Computer

• network: UoR LAN/eth0 (not over WiFi)
• CPU:

     *-cpu
          product: Intel(R) Core(TM) i5-6200U CPU @ 2.30GHz
          vendor: Intel Corp.
          physical id: 1
          bus info: cpu@0
          size: 2303MHz
          capacity: 2800MHz
          width: 64 bits

Test code:

import os
import numpy as np


from activestorage.active import Active


S3_ACTIVE_URL_Bryan = "https://192.171.169.248:8080"
S3_BUCKET = "bnl"

def gold_test():
    """Run somewhat as the 'gold' test."""
    storage_options = {
        'key': "f2d55c6dcfc7618b2c34e00b58df3cef",
        'secret': "$/'#M{0{/4rVhp%n^(XeX$q@y#&(NM3W1->~N.Q6VP.5[@bLpi='nt]AfH)>78pT",
        'client_kwargs': {'endpoint_url': "https://uor-aces-o.s3-ext.jc.rl.ac.uk"},
    }
    active_storage_url = "https://192.171.169.248:8080"
    bigger_file = "ch330a.pc19790301-bnl.nc"

    test_file_uri = os.path.join(
        S3_BUCKET,
        bigger_file
    )
    print("S3 Test file path:", test_file_uri)
    active = Active(test_file_uri, 'UM_m01s16i202_vn1106', storage_type="s3",
                    storage_options=storage_options,
                    active_storage_url=active_storage_url)
    # old test with 3GB file
    # active2 = Active(test_file_uri, 'm01s06i247_4', storage_type="s3",
    #                 storage_options=storage_options,
    #                 active_storage_url=active_storage_url)

    active._version = 1
    active._method = "min"

    result = active[:]
    # result = active[0:3, 4:6, 7:9]  # standardized slice

    print("Result is", result)
    return result

Kerchunk is restricted to Dataset of interest:

Looking only at a single Dataset <HDF5 dataset "UM_m01s16i202_vn1106": shape (40, 1920, 2560), type "<f4">

Chunks

Both Kerchunk and Pyfive send variable (give or take 5 or 10) numbers of chunks to Reductionist; order of magnitude is 3360 chunks.

Kerchunk-based Pipeline

Result is 4677.8594 (stable)

• 18.03user 2.24system 1:37.77elapsed 20%CPU (0avgtext+0avgdata 202112maxresident)k
• 20.00user 2.02system 1:35.60elapsed 23%CPU (0avgtext+0avgdata 203124maxresident)k
• 19.64user 2.26system 1:34.86elapsed 23%CPU (0avgtext+0avgdata 201880maxresident)k
• 20.95user 2.43system 1:34.75elapsed 24%CPU (0avgtext+0avgdata 200884maxresident)k
• 14.94user 1.49system 1:34.19elapsed 17%CPU (0avgtext+0avgdata 201932maxresident)k
• 15.47user 1.72system 1:47.83elapsed 15%CPU (0avgtext+0avgdata 203052maxresident)k
• 20.04user 2.19system 1:33.50elapsed 23%CPU (0avgtext+0avgdata 202192maxresident)k
• 19.73user 2.08system 1:35.95elapsed 22%CPU (0avgtext+0avgdata 202144maxresident)k
• 20.65user 2.44system 1:31.98elapsed 25%CPU (0avgtext+0avgdata 200952maxresident)k

Kerchunk indexing and JSON file writing times:

• Time to Kerchunk and write JSON file 21.811710596084595
• Time to Kerchunk and write JSON file 20.934044361114502
• Time to Kerchunk and write JSON file 21.715813636779785
• Time to Kerchunk and write JSON file 21.793660879135132

Pyfive-based pipeline

Result is 4677.8594 (stable)

• 21.54user 3.07system 1:22.10elapsed 29%CPU (0avgtext+0avgdata 195224maxresident)k
• 21.28user 2.79system 1:19.94elapsed 30%CPU (0avgtext+0avgdata 196944maxresident)k
• 21.47user 2.73system 1:25.87elapsed 28%CPU (0avgtext+0avgdata 198084maxresident)k
• 21.05user 2.93system 1:35.86elapsed 25%CPU (0avgtext+0avgdata 197568maxresident)k
• 21.45user 2.78system 1:30.15elapsed 26%CPU (0avgtext+0avgdata 197820maxresident)k

Sliced Kerchunk (slice [0:3, 4:6, 7:9])

• Time to Kerchunk and write JSON file 21.60s; 27s TOTAL
• Time to Kerchunk and write JSON file 22.16s; 27s TOTAL
• Time to Kerchunk and write JSON file 21.15s; 27s TOTAL
• Time to Kerchunk and write JSON file 22.61s; 28s TOTAL

Sliced Pyfive (slice [0:3, 4:6, 7:9])

• 14s TOTAL
• 13s TOTAL
• 12s TOTAL
• 13.4s TOTAL

[valeriupredoi]

On JASMIN/sci2

---

CPU:

     *-cpu
          product: Intel(R) Xeon(R) Gold 6348 CPU @ 2.60GHz
          vendor: Intel Corp.
          vendor_id: GenuineIntel
          physical id: 1
          bus info: cpu@0
          version: 6.58.0
          width: 64 bits

Kerchunk-based Pipeline

Result is 4677.8594 (stable)

• 7.90user 1.58system 1:32.65elapsed 10%CPU (0avgtext+0avgdata 193264maxresident)k
• 7.70user 1.54system 1:34.54elapsed 9%CPU (0avgtext+0avgdata 193440maxresident)k
• 7.43user 1.57system 1:30.73elapsed 9%CPU (0avgtext+0avgdata 194988maxresident)k

Kerchunk indexing and JSON file writing times:

• Time to Kerchunk and write JSON file 20.565886735916138
• Time to Kerchunk and write JSON file 23.35599684715271
• Time to Kerchunk and write JSON file 21.062350749969482

Pyfive-based pipeline

• 7.25user 1.59system 1:16.66elapsed 11%CPU (0avgtext+0avgdata 188192maxresident)k
• 7.38user 1.57system 1:20.16elapsed 11%CPU (0avgtext+0avgdata 188392maxresident)k
• 7.36user 1.62system 1:16.32elapsed 11%CPU (0avgtext+0avgdata 182540maxresident)k
• 7.64user 1.71system 1:13.63elapsed 12%CPU (0avgtext+0avgdata 194316maxresident)k

[valeriupredoi]

Question no 1

• former 2.8G file dims: 30 x 39 x 325 x 432 = 1.6e8 elements -> Time to Kerchunk and write JSON file 0.097s
• new 30G test file dims: 40 x 1920 x 2560 = 2.0e8 elements -> Time to Kerchunk and write JSON file 21.60s

Answer

@bnlawrence suggests chunking, and he is correct: 2.8G file field has 30 chunks, the other field has 3400 chunks -> here's the penalty factor right there!

[valeriupredoi]

Use -def file (64 HDF5 chunks)

Kerchunk-based pipeline

My computer (UoR network etc)

• 3.51user 0.52system 0:09.65elapsed 41%CPU (0avgtext+0avgdata 156440maxresident)k
• 3.46user 0.52system 0:10.52elapsed 37%CPU (0avgtext+0avgdata 157688maxresident)k
• 3.31user 0.55system 0:09.39elapsed 41%CPU (0avgtext+0avgdata 149540maxresident)k

Kerchunk indexing and JSON file writing times:

• Time to Kerchunk and write JSON file 2.3882689476013184
• Time to Kerchunk and write JSON file 2.316840410232544
• Time to Kerchunk and write JSON file 2.4828641414642334

Time before going into Reductionist:

• 1.43user 0.28system 0:06.88elapsed 25%CPU (0avgtext+0avgdata 102108maxresident)k
• 1.44user 0.29system 0:06.82elapsed 25%CPU (0avgtext+0avgdata 102236maxresident)k
• 1.47user 0.23system 0:06.83elapsed 25%CPU (0avgtext+0avgdata 101864maxresident)k

Pyfive-based pipeline

• 3.55user 0.85system 0:07.71elapsed 57%CPU (0avgtext+0avgdata 153428maxresident)k
• 3.49user 0.67system 0:07.08elapsed 58%CPU (0avgtext+0avgdata 146776maxresident)k
• 3.76user 0.74system 0:07.72elapsed 58%CPU (0avgtext+0avgdata 148100maxresident)k
• 3.70user 0.71system 0:07.64elapsed 57%CPU (0avgtext+0avgdata 153924maxresident)k

Time before going into Reductionist:

• 1.48user 0.31system 0:03.54elapsed 50%CPU (0avgtext+0avgdata 93096maxresident)k
• 1.43user 0.35system 0:03.56elapsed 50%CPU (0avgtext+0avgdata 88560maxresident)k
• 1.47user 0.31system 0:04.34elapsed 41%CPU (0avgtext+0avgdata 88752maxresident)k
• 1.43user 0.35system 0:03.59elapsed 49%CPU (0avgtext+0avgdata 93080maxresident)k

[valeriupredoi]

so it's starting to look like this:

Kerchunk-based pipeline

Kerchunk indexer:

• 1.6e8 elements 30 chunks = 0.1s - scale to 2e8 - > 0.14s -> 0.04s/chunk
• 2.0e8 elements 64 chunks = 2.5s .....................................-> 0.04s/chunk
• 2.0e8 elements 3400 chunks = 22.0s ....................................-> 0.006s/chunk

To (network) and at Reductionist

• 1.6e8 elements 30 chunks = 2s -> 0.07s/chunk
• 2.0e8 elements 64 chunks = 3s -> 0.05s/chunk
• 2.0e8 elements 3400 chunks = 75.0s -> 0.02s/chunk

Total time

• 1.6e8 elements 30 chunks = 4s -> 0.13s/chunk
• 2.0e8 elements 64 chunks = 10s -> 0.12s/chunk
• 2.0e8 elements 3400 chunks = 100s -> 0.02s/chunk