About the App. App name: gdal; App description: GDAL: Geospatial Data Abstraction Library; App website: Not Available; Install the App. Press Command+Space and type Terminal and press enter/return key.
Do not pip install gdal, do not install GDAL inside a virtual-env. Instead, use Conda.
These are my instructions on how to install GDAL using Conda on a Mac. Before we dive in, let me explain why I am writing this guide. GDAL stands for the “Geospatial Data Abstraction Library” and it is released by the Open Source Geospatial Foundation. For Python, the GDAL package is released with a package called
osgeo
as well. And as it happens, I need both for a project I’m doing. And I want them neatly wrapped up inside a virtual environment.I started by trying to use pip and virtual-env. This obscure how-to (and first post on Medium!) comes to you after hours of googling and trying to debug the errors I encountered along the way. In the end, I shifted gears, tried something new and switched over to Conda. This was the solution to all my problems. OK, except a few, but I fixed those too.
Conda is great because it’s a package manager like pip, but it also manages your virtual environments like virtual-env does. Except it does both way better and it’s a joy to use. Treat yo self and switch to Conda.
I wanted to spare you the trouble I experienced so I wrote up the following instructions.
Environment:
A quick note about the environment I’m working in:
Mac OS 10.12.6 Sierra
Miniconda2 for Python 2 (Conda 4.3.30) info
Python 2.7.14
optional python packages:
Jupyter notebook, installs with$ conda install jupyter
nb_conda, makes Jupyter play nice with Conda,$ conda install nb_conda
If you are working in a different development environment, your mileage may vary.
And now, the instructions that I myself needed hours ago…
How to Install GDAL ? using Conda ? on a Mac ?
This tutorial assumes you have Conda already installed and a Conda environment already created. Instructions here and here, respectively.
Step 1: Activate your Conda environment ?
Open up Terminal, run this:
For me, [yourEnvironmentName] =
geoenv
.(We can deactivate with the command
$ source deactivate
.)Step 2: Ok, now we get to install GDAL. ?
Still in Terminal, run this command:
Here’s what I got:
Great. As it turns out, for the osgeo subpackage to work, we actually need the dependency
jpeg version 8
, rather than 9
. You can read more about how I came to that conclusion towards the end of this post, under #Diagnosing.For now, all you need to do is run this:
The “-f” flag forces the install (which is really a downgrade of the the jpeg module).
OK, we should have a working version of GDAL now! Let’s just test it to make sure.
Step 3: Test the installation ?
You can do this in the command line or in a Jupyter notebook. Since I want to make sure gdal will work in Jupyter later, I’m going to test there.
To open a new Jupyter Notebook ?, go back to Terminal, run this command:
This command will open up a new tab in your internet browser with the Jupyter Notebook file viewer. Navigate to the directory where you wish to save your notebook. Now, we want to start a new notebook. Go to the upper righthand corner, click “New”.
Make sure to choose the Conda environment you’ve been working with as the Python Kernel.
Let’s go ahead and test! Run these commands in the notebook.
The help for gdal works, so we’re off to the races.
Success! ?
We’ve finally got GDAL installed as well as osgeo. Everything is working (for now). This somewhat lengthy post was a joy to write, as this problem caused me innumerable hours of strife. I hope to save you from the same. Thanks in advance for your claps ?? Let me know if something needs an edit or clarification. With that, I’m off to explore graphs with
networkx
!With ❤︎,
Filip
p.s.
#Motivation for this post
I generally like to use virtual environments on projects to keep things organized. First, I tried to install GDAL inside a python virtual-env which was a huge fail. There are instructions out there how to do that for Windows and Ubuntu, but I couldn’t get it to work for Mac. Virtual-env was more like virtual-enemy. Some folks on StackOverflow suggested to use Conda instead. I ran into a few snags anyways, so I decided to publish these instructions how to Install GDAL using Python/Conda on Mac. Dear reader, I hope this guide saves you some time.
#What am I using GDAL for
I need GDAL for a very particular reason. It’s required for and a dependency of the
read_shp()
function of the networkx Python module. That function reads in an ESRI shapefile (geospatial data) an converts it into a network/graph object. Obviously, you might need GDAL for something else.To install, I tried using
pip install gdal
inside a Python virtual environment (a.k.a. virtual-env) at first. That failed. I guess you could say it was only a pip dream, sigh. Or maybe it had something to do with having QGIS via Kyngchaos installed. That distribution includes GDAL not as a Python package, but as a Framework.Anyways, the bottom line is that I still needed GDAL to work inside a Python virtual environment.
#Diagnosing the Conda install issue:
It was not possible for me to get GDAL installed inside a virtual-env using pip. That’s why I switched to Conda.
When running the install in Conda, I ran into a few issues. Simply running the
read_shp
function from networkx
was giving me a generic error, much like it was in virtual-env.ImportError: read_shp requires OGR:
In the screenshot you can see that the code requires
from osgeo import ogr
which is actually included as part of the GDAL module.So when we try to
import gdal
, we can see what’s actually happening:Library not loaded: @rpath/libjpeg.8.dylib
The jpeg8 library is not loading. To investigate, we can check what packages conda has installed:
$ conda list
Moreover, when I uninstall and reinstall only
gdal
, it actually becomes evident that gdal itself updates jpeg to version 9, only to break later.The fix is to simply downgrade jpeg 9 to jpeg 8 after installing gdal. You can find the recipe for that in Step 3 of the #Instructions above. Thanks!
Sources I used:
- egayer’s comment in this thread on the gdal GitHub
- I’m not including my crazy, exhaustive searches for anything related to “pip install GDAL” or “GDAL python install mac virtual-env” in this list. Bless your heart if you try to go that path.
We automate wheel building using this custom github repository that builds onthe travis-ci OSX machines and the travis-ci Linux machines.
The travis-ci interface for the builds ishttps://travis-ci.org/MacPython/gdal-wheels
The driving github repository ishttps://github.com/MacPython/gdal-wheels
How it works
The wheel-building repository:
- does a fresh build of any required C / C++ libraries;
- builds a GDAL wheel, linking against these fresh builds;
- processes the wheel using delocate (OSX) or auditwheel
repair
(Manylinux1).delocate
andauditwheel
copy the required dynamiclibraries into the wheel and relinks the extension modules against thecopied libraries; - uploads the built wheels to http://wheels.scipy.org (a Rackspace containerkindly donated by Rackspace to scikit-learn).
The resulting wheels are therefore self-contained and do not need any externaldynamic libraries apart from those provided as standard by OSX / Linux asdefined by the manylinux1 standard.
The
.travis.yml
file in this repository has a line containing the API keyfor the Rackspace container encrypted with an RSA key that is unique to therepository - see http://docs.travis-ci.com/user/encryption-keys. Thisencrypted key gives the travis build permission to upload to the Rackspacedirectory pointed to by http://wheels.scipy.org.Triggering a build
You will likely want to edit the
.travis.yml
file to specify theBUILD_COMMIT
before triggering a build - see below.You will need write permission to the github repository to trigger new buildson the travis-ci interface. Contact us on the mailing list if you need this.
You can trigger a build by:
- making a commit to the gdal-wheels repository (e.g. with gitcommit --allow-empty); or
- clicking on the circular arrow icon towards the top right of the travis-cipage, to rerun the previous build.
In general, it is better to trigger a build with a commit, because this makesa new set of build products and logs, keeping the old ones for reference.Keeping the old build logs helps us keep track of previous problems andsuccessful builds.
Which GDAL commit does the repository build?
The gdal-wheels repository will build the commit specified in the
BUILD_COMMIT
at the top of the .travis.yml
file. This can be anynaming of a commit, including branch name, tag name or commit hash.Uploading the built wheels to pypi
Be careful, http://wheels.scipy.org points to a container on a distributedcontent delivery network. It can take up to 15 minutes for the new wheel fileto get updated into the container at http://wheels.scipy.org.
The same contents appear athttps://3f23b170c54c2533c070-1c8a9b3114517dc5fe17b7c3f8c63a43.ssl.cf2.rackcdn.com;you might prefer this address because it is https.
When the wheels are updated, you can download them to your machine manually,and then upload them manually to pypi, or by using twine. You can also use ascript for doing this, housed at :https://github.com/MacPython/terryfy/blob/master/wheel-uploader
For the
wheel-uploader
script, you'll need twine and beautiful soup 4.You will typically have a directory on your machine where you store wheels,called a wheelhouse. The typical call for wheel-uploader would thenbe something like:
where:
-u
gives the URL from which to fetch the wheels, here the https address,for some extra security;-s
causes twine to sign the wheels with your GPG key;-v
means give verbose messages;-t all
downloads / uploads built wheels at this URL for all platforms(macOS, manylinux, windows - if present);-w ~/wheelhouse
means download the wheels from to the local directory~/wheelhouse
.
scikit_image
is the root name of the wheel(s) to download / upload, and2.2.1
is the version to download / upload.Eclipse For Mac
In order to upload to the PyPI server, you will need something like this inyour
~/.pypirc
file:Python
So, in this case,
wheel-uploader
will download all wheels starting withgdal-2.2.1-
from http://wheels.scipy.org to ~/wheelhouse
,then upload them to PyPI.Download Python For Mac Os
Of course, you will need permissions to upload to PyPI, for this to work.