CMake Error at /home/myuan/.pyenv/versions/3.11.3/lib/python3.11/site-packages/cmake/data/share/cmake-3.26/Modules/CMakeDetermineCompilerId.cmake:751 (message): Compiling the CUDA compiler identification source file "CMakeCUDACompilerId.cu" failed.
Address Length Type String .rdata:006E729C 00000028 C BVerifyInstalledFiles: bad directory %s .rdata:006E72C4 00000037 C BVerifyInstalledFiles: %s is %lld bytes, expected %lld .rdata:006E72FC 00000026 C BVerifyInstalledFiles: can't read %s\n .rdata:006E7324 0000003F C BVerifyInstalledFiles: bad CRC on %s (%x expected, %x actual)\n
I’ve created a package (czi-shader) to achieve what is mentioned in the title.
Introduction
Most of the lab’s slices are scanned using Zeiss microscopes. Previously, when working with CZI files outside Zen, you either had to export them within the software or directly convert them to pseudocolor using Python. But how are the appropriate colors assigned to different fluorescence channels when displaying the images?
Principles
After experimentation, the process of going from raw light to the final display can be divided into two steps: single-channel coloring and multi-channel merging.
Single-Channel Coloring
CTB488 is usually specified as fluorescence green, with RGB saturation values of (0, 255, 91), which translates to (141, 100%, 50%) in the HSL color space. Let the actual brightness obtained from scanning be denoted as $l_r$. The resulting color is calculated as follows:
$$ \begin{equation} \begin{aligned} H &= 141 \ S &= 100% \ L &= \frac{l_r - \text{low}}{\text{high} - l_r} * 50% \end{aligned} \end{equation} $$
This process is applied to each channel, resulting in multiple (h, w, 3) matrices, where 3 represents the three components of the HSL color space.
Multi-Channel Merging
Zen’s merging is done in the RGB space. The HSL space images obtained above are converted to RGB and then summed and clipped. The equivalent numpy code is as follows:
Now that we have the calculation method, where can we find the raw data for the channels? Based on experiments, in the CZI metadata under .//DisplaySetting/Channels/, a sample data entry looks like this:
The data format is self-explanatory, but parsing this XML and converting it into appropriate data for coloring the raw light can be repetitive. Therefore, I have developed a Python package that allows you to perform these operations directly.
1 2 3 4 5 6 7 8
import cv2 from czi_shader import CZIChannel, shading_czi
p = '/mnt/inner-data/sc-C057-146-O4213.czi' [print(c) for c in CZIChannel.from_czi(p)]
res = shading_czi(p, scale_factor=0.01) cv2.imwrite(p + '.png', cv2.cvtColor(res, cv2.COLOR_RGB2BGR))
I admire those powerful tools that could do a lot with just a few megabytes in the past. Besides, the performance requirements for numpy in this program are not high. So I tried to remove this dependency. There are many discussions online about packaging and reducing package size, but they only replace mkl with openblas. However, openblas is still quite large.
After searching, this site provides builds without openblas and mkl, but the last update was in 2022, and the version is still 1.22.4, which is a bit old. Since third-party builds already exist, I guessed this might not be too difficult.
Preparation
Compiler
On Windows, just downloadMicrosoft C++ Build Tools, start it and select Desktop development with C++. Wait for it to finish and you are good to go. No need to set environment variables as numpy will handle it later.
Source Code
1 2 3
git clone --recurse-submodules https://github.com/numpy/numpy.git cd numpy git checkout maintenance/1.24.x # or 1.25.x, doesn't matter
Disable openblas
In numpy/distutils/, add a file site.cfg with the following content:
Later we will run commands in the .#env virtual environment by default.
Build
For <=1.24.x, you can build with bare msvc like this:
1
python setup.py build -j 16
However, for 1.25+, this will silently fail on Windows by generating compilation commands longer than 32768 lines.
A viable solution is to switch to using cibuildwheel.
Modify pyproject.toml, find [tool.cibuildwheel], and comment out before-build, before-test, test-command by adding # in front or just delete them. It should look like:
The before-build script is for downloading openblas. The two test steps don’t work for me.
Then build in PowerShell:
1 2 3
$env:CIBW_BUILD="cp311-win_amd64"# cp311 means CPython3.11, change to other versions if needed $env:CIBW_ENVIRONMENT="NPY_USE_BLAS_ILP64=0"# Don't know why it doesn't read site.cfg, use env var to emphasize cibuildwheel --platform windows
If everything goes well, you should see a whl file generated under wheelhouse:
1 2
-rwxrwxrwx 1 root root 6.1M Aug 7 23:37 wheelhouse/numpy-1.25.2-cp310-cp310-win_amd64.whl -rwxrwxrwx 1 root root 6.1M Aug 7 23:44 wheelhouse/numpy-1.25.2-cp311-cp311-win_amd64.whl
> python -c "import numpy as np; np.show_config(); print(np.__version__)" blas_armpl_info: NOT AVAILABLE blas_mkl_info: NOT AVAILABLE blas_ssl2_info: NOT AVAILABLE blis_info: NOT AVAILABLE openblas_info: NOT AVAILABLE accelerate_info: NOT AVAILABLE atlas_3_10_blas_threads_info: NOT AVAILABLE atlas_3_10_blas_info: NOT AVAILABLE atlas_blas_threads_info: NOT AVAILABLE atlas_blas_info: NOT AVAILABLE blas_info: NOT AVAILABLE blas_src_info: NOT AVAILABLE blas_opt_info: NOT AVAILABLE lapack_armpl_info: NOT AVAILABLE lapack_mkl_info: NOT AVAILABLE lapack_ssl2_info: NOT AVAILABLE openblas_lapack_info: NOT AVAILABLE openblas_clapack_info: NOT AVAILABLE flame_info: NOT AVAILABLE atlas_3_10_threads_info: NOT AVAILABLE atlas_3_10_info: NOT AVAILABLE atlas_threads_info: NOT AVAILABLE atlas_info: NOT AVAILABLE lapack_info: NOT AVAILABLE lapack_src_info: NOT AVAILABLE lapack_opt_info: NOT AVAILABLE numpy_linalg_lapack_lite: language = c define_macros = [('HAVE_BLAS_ILP64', None), ('BLAS_SYMBOL_SUFFIX', '64_')] Supported SIMD extensions in this NumPy install: baseline = SSE,SSE2,SSE3 found = SSSE3,SSE41,POPCNT,SSE42,AVX,F16C,FMA3,AVX2 not found = AVX512F,AVX512CD,AVX512_SKX,AVX512_CLX,AVX512_CNL,AVX512_ICL 1.25.2
The difference is about 35 MiB, quite significant. Removing openblas, my tool can fit in a single 15 MiB file. Although it’s still some way from my goal of a few megabytes, this is Python after all, so I’m quite satisfied.
Misc
The commit I built 1.25.x on is ea677928332c37e8052b4d599bf6ee52cf363cf9. Reset to it if yours is different.
My Windows version is 22H2 19045.3271
The pure build takes around 2 minutes on my trashy E5-2678.
> python -c "import numpy as np; np.show_config(); print(np.__version__)" blas_armpl_info: NOT AVAILABLE blas_mkl_info: NOT AVAILABLE blas_ssl2_info: NOT AVAILABLE blis_info: NOT AVAILABLE openblas_info: NOT AVAILABLE accelerate_info: NOT AVAILABLE atlas_3_10_blas_threads_info: NOT AVAILABLE atlas_3_10_blas_info: NOT AVAILABLE atlas_blas_threads_info: NOT AVAILABLE atlas_blas_info: NOT AVAILABLE blas_info: NOT AVAILABLE blas_src_info: NOT AVAILABLE blas_opt_info: NOT AVAILABLE lapack_armpl_info: NOT AVAILABLE lapack_mkl_info: NOT AVAILABLE lapack_ssl2_info: NOT AVAILABLE openblas_lapack_info: NOT AVAILABLE openblas_clapack_info: NOT AVAILABLE flame_info: NOT AVAILABLE atlas_3_10_threads_info: NOT AVAILABLE atlas_3_10_info: NOT AVAILABLE atlas_threads_info: NOT AVAILABLE atlas_info: NOT AVAILABLE lapack_info: NOT AVAILABLE lapack_src_info: NOT AVAILABLE lapack_opt_info: NOT AVAILABLE numpy_linalg_lapack_lite: language = c define_macros = [('HAVE_BLAS_ILP64', None), ('BLAS_SYMBOL_SUFFIX', '64_')] Supported SIMD extensions in this NumPy install: baseline = SSE,SSE2,SSE3 found = SSSE3,SSE41,POPCNT,SSE42,AVX,F16C,FMA3,AVX2 not found = AVX512F,AVX512CD,AVX512_SKX,AVX512_CLX,AVX512_CNL,AVX512_ICL 1.25.2
Sometimes you need dynamic enumerations, such as some selector options from database, but you will find that it seems like an the enumeration can only be modified at app startup. Try this:
1 2 3 4 5 6 7 8 9 10 11
myuan@jxtkfuwuqi ~> curl 10.242.155.222:8000 {"detail":[{"loc":["query","e"],"msg":"field required","type":"value_error.missing"}]}⏎ myuan@jxtkfuwuqi ~> curl 10.242.155.222:8000/rand-enum {"message":{"0-7":"value-85","1-8":"value-6","2-5":"value-10","3-5":"value-99","4-4":"value-14"}}⏎ myuan@jxtkfuwuqi ~> curl '10.242.155.222:8000?e=value-10' {"e":"value-10"}⏎ myuan@jxtkfuwuqi ~> curl '10.242.155.222:8000?e=value-9' {"detail":[{"loc":["query","e"],"msg":"value is not a valid enumeration member; permitted: 'value-85', 'value-6', 'value-10', 'value-99', 'value-14'","type":"type_error.enum","ctx":{"enum_values":["value-85","value-6","value-10","value-99","value-14"]}}]}⏎ myuan@jxtkfuwuqi ~> curl '10.242.155.222:8000?e=value-11' {"detail":[{"loc":["query","e"],"msg":"value is not a valid enumeration member; permitted: 'value-85', 'value-6', 'value-10', 'value-99', 'value-14'","type":"type_error.enum","ctx":{"enum_values":["value-85","value-6","value-10","value-99","value-14"]}}]}⏎
Enumeration constraints are still in effect while the enumeration is modified
