# Setting up the environment

## General overview

Most library dependencies can be installed from PyPI using pip. There are different target requirements files.
Using a dedicated virtualenv is recommended. To install everything for a target run
`pip install -r requirements/<target>.<platform>.txt` or `pip-sync requirements/<target>.<platform>.txt`. The
latter will remove not needed packages.
These will only work if all packages are available for the current platform specifically. For those that have to be
installed from source packages, all build dependencies - like C compiler - must be set up, and the development
libraries must be available globally. Additionally, the `clarabot-crypto` must be available and the `clarabot-testing`
must be available for development.

Running Nano from the source or packaging it will fetch the Solr and Tika dependencies automatically. When those are
updated (or downloaded for the first time) the jdeps tool generates their java dependencies for jlink to compile a
minimal JRE for running and distribution. This takes several minutes. Gradle is used for building a Java worker
for thumbnail generation, but that is automatically set up with the Gradle Wrapper script from the repository.

System requirements:
    - Python 3.10
    - wxWidgets 3+ (compatible with wxPython 4.2)
    - protobuf library
    - dbus library (linux)
    - OpenJDK 11

### Windows 10+

Python is compiled with MSVC for Windows. We need to install it (and the Windows 10 SDK) for some binary extensions
that are not distributed in pre-built packages. For details see https://wiki.python.org/moin/WindowsCompilers .
Installing the integrated cmake as well is recommended as we'll need it.

**gcld3** needs Google's protobuf library to be built. The protobuf-compiler is available pre-built, but we can't use
it, since we also need the library bindings. Follow the general instructions for building the dependency available at
https://github.com/protocolbuffers/protobuf/blob/master/cmake/README.md . Options to use: x64 arch, build from the
latest stable git checkout (3.15.6 at the time of writing), release build, use dynamic linking (we have to for python),
use cmake to create nmake configuration, make, check, install. (zlib is not needed)
Copy the `bin`, `include` and `lib` to the virtualenv: `Scripts`, `include` and `libs`. The upstream package's build
script is broken on Windows. Install from this fork:
`pip install git+https://github.com/Traktormaster/cld3@fix-windows#egg=gcld3`
After installation the `libprotobuf.dll` has to be copied next to the pyd extension to be automatically loaded at
runtime. It is at `<venv>\lib\site-packages\gcld3`. See https://docs.python.org/3/library/os.html#os.add_dll_directory
for more information.

## Setup for packaging

1. Firstly clone the branch of `clarabot-nano` with which you want to work.
   Check in the `requirements/base.<platform>.txt` for the required version of `clarabot-crypto`.
   
2. Clone each of the dependent repositories, checkout the required version's tag and build their source-distribution package. Example:
   ```
   git clone git@dc.clarabot.zrt:Clarabot/clarabot-crypto.git
   cd clarabot-crypto
   git checkout v0.8.6
   python setup.py sdist
   ```
   Package should be in `dist` folder.

3. Create a virtualenv to work in `python -m venv <name>` and activate it `source <name>/bin/activate`.
   
   Install some utilities, then the custom packages, then sync the environment:
   ```
   pip install wheel
   pip install pip-tools
   pip install clarabot-crypto-xxx.tar.gz
   pip-sync clarabot-nano/requirements/pkg.<platform>.txt
   ```

4. Build the frozen app and the package.

   In the `clarabot-nano` repository. (add it to `PYTHONPATH` envvar if necessary)

   ```
   python -m dev_tools.build
   ```
   
    NOTE: If you see `Exception: Site cleared, try again...` just try again...

## Building for Linux on CentOS7

Install CentOS 7 with `GNOME Desktop` base environment, select `Development Tools`

1. Add EPEL and install dependencies
   ```
   sudo yum install epel-release
   sudo yum repolist && sudo yum update
   sudo yum install git nano mc libffi libffi-devel openssl11 openssl11-libs openssl11-devel sqlite sqlite-devel readline readline-devel libuuid libuuid-devel uuid uuid-devel protobuf protobuf-devel wxGTK wxGTK-devel wxGTK3 wxGTK3-devel patchelf
   ```

2. Download an adoptium JDK (Java in repos is broken) then extract it into `/opt`
   ```
   sudo alternatives --install /usr/bin/java java /opt/jdk-11.0.17+8/bin/java 0
   sudo alternatives --config java
   ```
   
   Then the building user: `nano ~/.bashrc`
   ```
   export PATH=/opt/jdk-11.0.17+8/bin
   ```

3. Build python
   - Download and extract desired source tar(https://github.com/python/cpython/tags)
      ```
      cd /tmp
      wget https://www.python.org/ftp/python/3.10.9/Python-3.10.9.tgz
      wget https://www.python.org/ftp/python/3.10.9/Python-3.10.9.tgz.asc
      gpg --keyserver keyserver.ubuntu.com --recv-keys 64E628F8D684696D
      gpg --verify Python-3.10.9.tgz.asc
      tar -xf Python-3.10.9.tgz
      cd Python-3.10.9
      ```

   - Configure and compile
      ```
      export CXXFLAGS="-std=c++11"
      export CFLAGS="-std=c++11 -march=core2 -mtune=corei7 $(pkg-config --cflags openssl11)"
      export LDFLAGS="$(pkg-config --libs openssl11)"
      ./configure --enable-shared --prefix=/opt/python310
      make -j${nproc}
      sudo make install
      ```
      NOTE: The std=c++11 options are necessary for gcld3 and wxPython 4.2 compilation. (the wef build of wxPython must be patched to include these flags as well around line 200)

   - Set LD path
      ```
      export LD_LIBRARY_PATH=/opt/python310/lib
      ```

4. Create venv and activate it
   ```
   /opt/python310/bin/python3 -m venv ~/.venv310
   source ~/.venv310/bin/activate
   pip install wheel
   pip install pip-tools
   ```

5. Get the Clarabot cryptography dependency from git to build its package
   ```
   git clone git@dc.clarabot.zrt:Clarabot/clarabot-crypto.git
   cd clarabot-crypto
   git checkout v1.0.1
   # optinally run the tests by pip-sync requirements/test.txt && ALLOW_TESTS_IMPORT=1 py.test
   python setup.py sdist
   pip install --no-deps dist/clarabot-crypto-1.0.1.tar.gz
   ```

6. Get the Clarabot testing dependency (if testing is desired) from git to build its package
   ```
   git clone git@dc.clarabot.zrt:Clarabot/clarabot-testing.git
   cd clarabot-testing
   git checkout v0.6.1
   python setup.py sdist
   pip install --no-deps dist/clarabot-testing-0.6.1.tar.gz
   ```

7. Get the Clarabot Nano from git. We install it in development/editable mode because tests need to be run.
   ```
   git clone git@dc.clarabot.zrt:Clarabot/clarabot-nano.git
   cd clarabot-nano
   git checkout v1.1.0
   ```

8. Run the tests. Remember to get the dev-server cert-config file to the `/tmp` directory and make sure "nano.clarabot.test" resolves and is accessible on the network.
   ```
   pip-sync requirements/test.linux.txt
   pip install -e .
   CBNANO_USER_DIR_OVERRIDE=devtest ALLOW_TESTS_IMPORT=1 py.test
   ```
   First run will take ages because it fetches & compiles the Java dependencies!

9. Switch the environment to package building.
   ```
   pip-sync requirements/pkg.linux.txt
   pip install -e .
   ```

10. Get appimagetool (if not already available).
    ```
    sudo mkdir /opt/appimage
    cd /opt/appimage
    sudo chmod +x appimagetool-x86_64.AppImage
    sudo mv appimagetool-x86_64.AppImage appimagetool.AppImage
    ```
    
    Then the building user: `nano ~/.bashrc`
    ```
    export PATH=/opt/appimage:${PATH}
    ```

11. Build clarabot-nano
    ```
    cd clarabot-nano
    python -m dev_tools.build
    ```
