{
 "cells": [
  {
   "attachments": {
    "06eda1e0-7e8e-45d2-8b32-7de9f64bccc9.png": {
     "image/png": "[encoded data, 62332 characters]"
    }
   },
   "cell_type": "markdown",
   "id": "217ea8bd",
   "metadata": {},
   "source": [
    "# xsar batch processing on datarmor (ifremer)\n",
    "\n",
    "> unlike others examples, this notebook is not executed by the documentation module, so there are no output cells.\n",
    "\n",
    "This notebook will show how to use `xsar` with `dask` to process many SAFEs, on datarmor.\n",
    "\n",
    "We will see how to convert some L1 SAFEs files to L1B netcdf files at 1000m resolution. \n",
    "\n",
    "Basically, to use your own processing function, you will have to adapt the `l1b` function defined below.\n",
    "\n",
    "## Prerequisite\n",
    "\n",
    "This example is fully documented, and shoul be used by new users that have never used xsar or dask in a jupyter notebook on datarmor.\n",
    "\n",
    "\n",
    "### Set up environment\n",
    "\n",
    "We will assume that xsar is not installed, and dask is not configured for datarmor.\n",
    "\n",
    "#### Get a node with internet access\n",
    "\n",
    "This is needed, because `conda` need more cpu/ram than available on the submit node, and an internet access.\n",
    "\n",
    "```\n",
    "ssh datarmor\n",
    "qsub -I -q ftp -l mem=16g,walltime=4:00:00\n",
    "```\n",
    "\n",
    "#### install environement\n",
    "\n",
    "\n",
    "Install (and update) xsar. More infos on  https://cyclobs.ifremer.fr/static/sarwing_datarmor/xsar/installing.html\n",
    "\n",
    "```\n",
    "conda create -n xsar\n",
    "conda activate xsar\n",
    "conda install -c conda-forge xsar 'python<3.10'\n",
    "pip install git+https://github.com/umr-lops/xsar.git\n",
    "pip install git+https://github.com/umr-lops/xsarsea.git\n",
    "```\n",
    "\n",
    "This is needed for coastlines, because datarmor nodes don't have internet access\n",
    "\n",
    "```\n",
    "cartopy_feature_download.py --output `python -c 'import cartopy ; print(cartopy.config[\"data_dir\"])'` physical\n",
    "```\n",
    "\n",
    "This is needed for https://datarmor-jupyterhub.ifremer.fr/\n",
    "\n",
    "```\n",
    "conda install -c conda-forge jupyterhub\n",
    "python -m ipykernel install --user\n",
    "```\n",
    "\n",
    "\n",
    "Install https://dask-hpcconfig.readthedocs.io/en/latest/\n",
    "\n",
    "```\n",
    "pip install git+https://github.com/umr-lops/dask-hpcconfig.git#egg=dask-hpcconfig\n",
    "```\n",
    "\n",
    "Enable dask-labextension, so the dask dashboard can be reached\n",
    "\n",
    "```\n",
    "pip install dask-labextension\n",
    "pip install ipywidgets\n",
    "```\n",
    "\n",
    "### download and execute this notebook\n",
    "\n",
    "Now , go to https://datarmor-jupyterhub.ifremer.fr, and choose 'jupyter lab'. Put 'xar' in the optionnal field, to use the previously created env.\n",
    "\n",
    "It better to use a 8Gb ram notebook or more to handle the dask scheduler.\n",
    "\n",
    "![image.png](attachment:06eda1e0-7e8e-45d2-8b32-7de9f64bccc9.png)\n",
    "\n",
    "Dowlnoad this notebook as ipynb, and open it jupyterhub. \n",
    "You are now ready to execute it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "729774cd",
   "metadata": {},
   "outputs": [],
   "source": [
    "import xsar\n",
    "import distributed\n",
    "import dask_hpcconfig\n",
    "import glob\n",
    "import pandas as pd\n",
    "import os\n",
    "import dask.dataframe as dd\n",
    "import time\n",
    "from tqdm.auto import tqdm, trange\n",
    "import numpy as np\n",
    "import traceback"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c25a9d9c-f99d-4fb7-b14e-c8aa8904df46",
   "metadata": {},
   "source": [
    "## Get input and outputs as rows of a pandas dataframe\n",
    "\n",
    "The first step is to build a `pandas.Dataframe` with input SAFEs and output paths. Basicallys, eachs rows will be fed to our processing function, in parallel.\n",
    "\n",
    "Elements size should be small, like filenames or very small python objects. Do not use here complex datatypes like `xsar.Sentinel1Meta` objects, or numpy array. (However, it's possible to use dask futures).\n",
    "\n",
    "It's important to be able to know if the processing is allready done, because we wan't to be able to skip what's allready done. \n",
    "\n",
    "It's a good practice to do that before calling the processing function, and a bad practice to do than *inside* the processing function, because a processing slot will be used on a worker to do nothing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bd7b6698",
   "metadata": {},
   "outputs": [],
   "source": [
    "# get input SAFEs, as a pandas dataframe\n",
    "df_safes = pd.DataFrame(glob.glob('/home/datawork-cersat-public/cache/project/mpc-sentinel1/data/esa/sentinel-1a/L1/IW/S1A_IW_GRDH_1S/2021/12*/*.SAFE'), columns=['safe'])\n",
    "# we just add an invalid SAFE, to be able to handle errors\n",
    "df_safes.loc[-1,'safe'] = 'error.SAFE'\n",
    "df_safes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c567261a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# compute out_path\n",
    "out_path_prefix = '%s/xsar_dask_demo' % os.environ['SCRATCH']\n",
    "os.makedirs(out_path_prefix, exist_ok=True)\n",
    "df_safes['out_path'] = df_safes['safe'].apply(lambda f: '%s/%s.nc' % (out_path_prefix, os.path.splitext(os.path.basename(f))[0]))\n",
    "df_safes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e3e34647",
   "metadata": {},
   "outputs": [],
   "source": [
    "# filter out_path that allready exists\n",
    "df_safes = df_safes[df_safes['out_path'].apply(lambda f: not os.path.exists(f))]\n",
    "df_safes"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7af3b0c-d8dd-482d-b880-2d216e3d9c25",
   "metadata": {},
   "source": [
    "## Main processing function\n",
    "\n",
    "the `l1b` function is the main processing function. It should take very small arguments size, like input file and output file. Do not use arguments with big size here (like xarray or complex objects)\n",
    "\n",
    "In this example, we just open the safe at 1000m resolution, and save it as a netcdf file.\n",
    "\n",
    "This is the function you will need to change for your own processing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "670d1520",
   "metadata": {},
   "outputs": [],
   "source": [
    "def l1b(safe, outfile):\n",
    "    ds = xsar.open_dataset(safe, resolution='1000m')['measurement'].ds\n",
    "    # make attributes to be str, so writable to file\n",
    "    to_str = ['start_date', 'stop_date', 'footprint']\n",
    "    for attr in to_str:\n",
    "        ds.attrs[attr] = str(ds.attrs[attr])\n",
    "    ds.to_netcdf(outfile)\n",
    "    return outfile"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6bccee48-a127-46a6-a13c-461f19ca9d3f",
   "metadata": {},
   "source": [
    "To process all SAFEs from `df_safes`, what we don't want to do is to use a sequential loop like\n",
    "\n",
    "```python\n",
    "for idx, safe in df_safes.iterrows():\n",
    "    print(l1b(*safe))\n",
    "```\n",
    "\n",
    "Because the processing whould be sequential (one safe is processed at a time), and it can take a long time.\n",
    "\n",
    "So what we want to do is to use dask to execute many `l1b` function in parallel."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3611804-15db-4c5f-9759-28c0aa12973d",
   "metadata": {},
   "source": [
    "## set up the dask cluster\n",
    "\n",
    "We use [dask-hpcconfig](https://dask-hpcconfig.readthedocs.io/en/latest/). See the doc to set up a different cluster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e952fee3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# see https://dask-hpcconfig.readthedocs.io/en/latest/ if you want to change the cluster config\n",
    "override = {\n",
    "    \"cluster.n_workers\": 28, \n",
    "    \"cluster.cores\": 1,\n",
    "    \"cluster.extra\": [\"--lifetime\", \"30m\", \"--lifetime-stagger\", \"10m\", \"--lifetime-restart\" ] # restart worker to clean state periodicaly\n",
    "}\n",
    "cluster = dask_hpcconfig.cluster('datarmor' , **override)\n",
    "cluster.scale(4)\n",
    "client = distributed.Client(cluster)\n",
    "client"
   ]
  },
  {
   "attachments": {
    "a88cb0c5-6117-4cca-bd0a-33b7a74fad73.png": {
     "image/png": "[encoded data, 64972 characters]"
    },
    "dfcc4b59-0f8d-4e99-bfe1-3d6574f6450d.png": {
     "image/png": "[encoded data, 111252 characters]"
    },
    "f1796c1e-fb37-4a66-8510-4e5b6d820921.png": {
     "image/png": "[encoded data, 197988 characters]"
    }
   },
   "cell_type": "markdown",
   "id": "81259c45-6426-4083-b6fd-2b4da997a5e5",
   "metadata": {},
   "source": [
    "If you click the above dashboard link, you should reach the dask status page that look like this:\n",
    "\n",
    "![image.png](attachment:a88cb0c5-6117-4cca-bd0a-33b7a74fad73.png)\n",
    "\n",
    "From now, the interresing parts are the 'workers' tab and the 'info' tab.\n",
    "\n",
    "### workers tab\n",
    "the workers tab should be something like\n",
    "\n",
    "![image.png](attachment:f1796c1e-fb37-4a66-8510-4e5b6d820921.png)\n",
    "\n",
    "If it's empty, the cluster is not allready instanciated (execute the cell bellow to wait for it)\n",
    "\n",
    "### info tab\n",
    "The info tab should be something like\n",
    "\n",
    "![image.png](attachment:dfcc4b59-0f8d-4e99-bfe1-3d6574f6450d.png)\n",
    "\n",
    "What's important here is the 'last seen' column. It should be above '1s'\n"
   ]
  },
  {
   "attachments": {
    "649971c1-6792-4ae8-a7c8-e8f2ff8dc46c.png": {
     "image/png": "[encoded data, 78316 characters]"
    }
   },
   "cell_type": "markdown",
   "id": "64a0e35b-e91e-49a7-875a-075ae9d3981b",
   "metadata": {},
   "source": [
    "## waiting for the cluster to be operationnal\n",
    "\n",
    "This should probably not needed, but we want to be sure that the cluster is ok.\n",
    "\n",
    "The main problem is that the `check` function that just do `import xsar` can be very long (up to 8 minutes).\n",
    "The issue is partialy solved by https://github.com/umr-lops/xsar/issues/65 , but it's probably due to a datarmor IO problem.\n",
    "\n",
    "The cluster should be up in 15s-60s, and the `import xsar` should take 30s-400s.\n",
    "\n",
    "The `import xsar` sometimes freeze the worker (probably because the GIL is not released). This can be show in the 'last seen' column of the 'info' tab as seen above. (If it's greater than 60s, the tab should turn red).\n",
    "\n",
    "![image.png](attachment:649971c1-6792-4ae8-a7c8-e8f2ff8dc46c.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "70e7d3d1",
   "metadata": {},
   "outputs": [],
   "source": [
    "while len(client.scheduler_info()['workers']) == 0:\n",
    "    print('waiting for cluster')\n",
    "    time.sleep(5)\n",
    "print('cluster is up. checking `import xsar`')\n",
    "\n",
    "def check():\n",
    "    import xsar\n",
    "    return True\n",
    "t0 = time.time()\n",
    "client.run(check)\n",
    "print('client checked in %d s' % (time.time() - t0 ))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee59b5c7-0797-4861-89a2-f7bde082e72c",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Split df_safes into smaller parts\n",
    "\n",
    "So we have `df_safes` that is a `pandas.Dataframe` object, with all SAFEs to be processed.\n",
    "\n",
    "We want to split this dataframe in smaller parts, and process those parts in parallel on different workers.\n",
    "\n",
    "To to so, we use `dask.dataframe`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "18f29854",
   "metadata": {},
   "outputs": [],
   "source": [
    "# we build a dask dataframe, with npartitions, so npartitions safes will be processed in parallel\n",
    "# npartitions depend on your processing and the workers count\n",
    "# a good starting value is to use the same number as the workers count\n",
    "# with heavy full res processing you will have to reduce it\n",
    "ddf_safes = dd.from_pandas(df_safes, npartitions=14)\n",
    "ddf_safes"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97306837-d755-4c24-81a2-36947795056a",
   "metadata": {},
   "source": [
    "## the batch processing function\n",
    "\n",
    "The `batch_processing` function will we executed on the workers. There will be as many `batch_processing` functions running in paralell as `npartitions`.\n",
    "\n",
    "Each functions will be fed with a partition of `ddf_safes`. A partition is a regular `pandas.Dataframe` object like `df_safes`, but with fewer rows.\n",
    "\n",
    "Basically, a minimal `batch_processing` function should look like this:\n",
    "\n",
    "```python\n",
    "def batch_processing(df_safes_part):\n",
    "    for _, safe_row in df_safes_part.iterrows():\n",
    "        safe, outfile = safe_row[['safe', 'out_path']]\n",
    "        l1b(safe, outfile)\n",
    "```\n",
    "\n",
    "the disadvantage of this method are:\n",
    "\n",
    "  * If the worker restart, and the function is retried, allready processed file will be reprocessed.\n",
    "  \n",
    "So we need to test if the outfile file exists before calling `l1b`\n",
    "\n",
    "  * Errors are not catched. If a `l1b` processing fail, remainings rows will be not processed.\n",
    "  \n",
    "We just need to surround the `l1b` call with a `try/except` block\n",
    "\n",
    "  * Errors messages are hidden in the workers logs\n",
    "  \n",
    "If a `l1b` processing fail, we need to know the error cause, and send it to the notebook.\n",
    "\n",
    "To be able to to that, we will use a `distributed.Queue` object, and will use it to send processing infos and error messages, as a dict like\n",
    "```\n",
    "{\n",
    "    'status': False,\n",
    "    'args': (safe, outfile),\n",
    "    'time': 0,\n",
    "    'error': \"\"\n",
    "}\n",
    "```\n",
    "\n",
    "  \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7910693c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# we set up a dask queue, so the batch_processing function, will be able to communicate processing infos to the notebook\n",
    "messages_queue = distributed.Queue('batch_processing')\n",
    "\n",
    "def batch_processing(df_safes_part, msg_queue=messages_queue):\n",
    "    res = []\n",
    "    for idx, safe_row in df_safes_part.iterrows():\n",
    "        safe, outfile = safe_row[['safe', 'out_path']]\n",
    "        # we need to re-check if outfile allready exist, because dask might have restarted the worker\n",
    "        # and we don't want the whole df_safes to be reprocessed\n",
    "        if os.path.exists(outfile):\n",
    "            res.append(outfile)\n",
    "            continue\n",
    "        \n",
    "        # we set up a dict that we will send to msg_queue \n",
    "        # it contains general processing info\n",
    "        message = {\n",
    "            'status': False,\n",
    "            'args': (safe, outfile),\n",
    "            'time': 0,\n",
    "            'error': \"\"\n",
    "        }\n",
    "        \n",
    "        # if a processing fail, we will retry\n",
    "        for retry in range(2):\n",
    "            # we enclose the processing in a try/except, so the worker won't be killed if error\n",
    "            try:\n",
    "                t1 = time.time()\n",
    "                # this is the real call to our processing function\n",
    "                out = l1b(safe, outfile)\n",
    "                elapsed = time.time() - t1\n",
    "                message['status'] = True\n",
    "                message['time'] = elapsed \n",
    "                break # process ok, exit the loop\n",
    "            except Exception as e:\n",
    "                # error while processing.\n",
    "                # we get the error message that will be sent to the queue\n",
    "                message['error'] = traceback.format_exc()\n",
    "        msg_queue.put(message)\n",
    "        res.append(outfile)\n",
    "    return res\n",
    "            \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46c230c3-73b3-4758-bed2-051fe5896124",
   "metadata": {},
   "source": [
    "## launch the distributed computation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a58c4618-3513-4d39-9610-55ee90a27de0",
   "metadata": {},
   "source": [
    "Now, we want to apply the `batch_processing` function to each partition of `dd_safes`.\n",
    "\n",
    "We use the `map_partitions` method. We use use a dummy `meta` keyword. What's important here is `str` (the output type of the `l1b` function)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0cff1cb4-6287-4ce6-bcee-e2f4a760d568",
   "metadata": {},
   "outputs": [],
   "source": [
    "res = ddf_safes.map_partitions(batch_processing, meta=('foo', str))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4c69de0-8618-47c6-8d9d-e8d549275855",
   "metadata": {},
   "source": [
    "At this stage, the computation has not yet started.\n",
    "\n",
    "One way to to it should be:\n",
    "\n",
    "```python\n",
    "res.compute()\n",
    "```\n",
    "\n",
    "But we won't be able to see messages from `messages_queue` and we won't see any progress information.\n",
    "\n",
    "we will instead use \n",
    "```\n",
    "res.persist()\n",
    "```\n",
    "\n",
    "and build a progress bar with `tqdm` that will wait for messages from `messages_queue`, and display status information.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "27ebd483",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "res.persist(retries=1)\n",
    "\n",
    "count = len(ddf_safes)\n",
    "\n",
    "pbar = trange(count,smoothing=0)\n",
    "elapsed = np.array([],dtype=float)\n",
    "for _ in pbar:\n",
    "    message = messages_queue.get()\n",
    "    if message['status']:\n",
    "        elapsed = np.append(elapsed,message['time'])\n",
    "        pbar.set_description('%03.0fs' % elapsed.mean())\n",
    "    else:\n",
    "        tqdm.write('ERROR: \"\\n%s\\n\" on args %s' % ( message['error'] , message['args']))"
   ]
  },
  {
   "attachments": {
    "0704e439-13d2-4838-aae1-1ef3d4207d65.png": {
     "image/png": "[encoded data, 20172 characters]"
    },
    "fd1fef54-cacd-4ee1-947c-b39dd0a8c2d3.png": {
     "image/png": "[encoded data, 108992 characters]"
    }
   },
   "cell_type": "markdown",
   "id": "48ddcdc0-a727-4478-836b-047b64c8bb9c",
   "metadata": {},
   "source": [
    "While processing occur, you should seel a progressbar like this:\n",
    "\n",
    "![image.png](attachment:0704e439-13d2-4838-aae1-1ef3d4207d65.png)\n",
    "\n",
    "18s is the mean time to process one SAFE, per worker. But as we have many workers, an new SAFE is processed every 1.39s.\n",
    "\n",
    "We have inserted an invalid 'error.SAFE' file. The error traceback should be displayed.\n",
    "\n",
    "While processing, the dask status dashboard should look like this:\n",
    "![image.png](attachment:fd1fef54-cacd-4ee1-947c-b39dd0a8c2d3.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8347db07-4604-49a7-aa43-17e07bdf7fd1",
   "metadata": {},
   "source": [
    "## End\n",
    "\n",
    "Once the processing is finished, processed files are in the output directory.\n",
    "\n",
    "We can now close the cluster.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "df35c0a8",
   "metadata": {},
   "outputs": [],
   "source": [
    "cluster.close()\n",
    "client.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d2d43ce0",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "84ebe89c",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "710632aa",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7b7ab189",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "env_xsar",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.15"
  },
  "nbsphinx": {
   "execute": "never"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
