How to get the location of the granules returned by a search?¶
Naiad only indexes the name of the granules, but not their location which may change as data are moved around.
However a simple mecanism is provided to get these locations when performing a search: the datastore configuration file.
FileLocator¶
A special class, naiad.utils.filelocator.FileLocator
is provided to compute
the file path of a granule given its name and the index identifier used in naiad
for the dataset it belongs to.
This class uses a configuration file, the datastore, to define the file repository structure for each dataset and how to retrieve from the granule file names the time value used to store this granule in its file repository structure. We assume here that granules are generally stored according to some time breakdown with intermediate repositories being year, month, day, day since year start, etc...
The FileLocator object will search the configuration file in the following order of locations:
- in the datastore file pathname specified in the call to
get_full_path
method. This is only available when using the FileLocator in a script.- in the datastore file pointed to by $NAIAD_DATASTORE environment variable.
- in a default datastore file stored in user home directory under
~/.naiad/
repository. You have to edit this file with your datastore configuration.
Configuration¶
Two configuration fields must be defined for each dataset indexed into Naiad:
storage_path
: the path and pattern (as in python datetime format string) of the directory structure of the datasets. The identifier of the dataset must match the one used to name the corresponding index in naiad.[storage_path] S3A_SL_1_RBT____NR=/my/data/store/%Y/%j/
time_extractor
: a python expression to extract a datetime object from a granule file name. It is used to infer the location of the file in its directory structure from the above storage path pattern, based on the time value used to decide in which date folder a granule should be stored (usually the sensing start time).[time_extractor] S3A_SL_1_RBT____NR=parser.parse(os.path.basename('$FILE').split('_')[-11])
Searching granules and getting full path¶
The FileLocator capabilities are used by Naiad command-line search tools.
naiad-search
can be called with the --full-path
argument. Instead of getting
just the granule names, you’ll get the full granule paths.
$ naiad-search iasi --full-path