Add docker container for druid #6896

donbowman · 2019-01-21T21:36:44Z

This container is an 'omnibus' (since there is such a high overlap with the various services). It includes all contrib extension as well as the msql connector.

It is intended to be run as docker run NAME SERVICE (e.g. docker run druid:latest broker)

Signed-off-by: Don Bowman db@donbowman.ca

This container is an 'omnibus' (since there is such a high overlap with the various services). It includes all contrib extension as well as the msql connector. It is intended to be run as `docker run NAME SERVICE` (e.g. docker run druid:latest broker) Signed-off-by: Don Bowman <db@donbowman.ca>

drcrallen

This is looking pretty good @donbowman

Would you mind posting in the dev list a question about if scripts that download GPL resources are ok in the repository?

.dockerignore

drcrallen · 2019-01-23T18:40:18Z

distribution/docker/Dockerfile

+ && ln -s /opt/apache-druid-${VER} /opt/druid
+
+RUN cd /opt/druid/extensions/mysql-metadata-storage \
+ && wget -O mysql-connector-java-5.1.38.jar http://central.maven.org/maven2/mysql/mysql-connector-java/5.1.38/mysql-connector-java-5.1.38.jar \


This might cause issues. Not sure what the limits of GPL stuff are, but having a script to download GPL components may or may not fly with Apache guidelines, will need confirmation.

If this can be added as another (optional) layer that will probably help things out

it is its own layer, but i don't really think it makes any difference. I'm not sure how i would make it optional, a sep build arg? a sep dockerfile?
the linux components (e.g. bash) are already gpl, so its not possible to have a container which doesn't have gpl.
and the gpl allows aggregation like this without attaching.
the other apache containers (e.g. tomcat, httpd, maven) all have gpl components in them.

distribution/docker/Dockerfile

drcrallen · 2019-01-23T18:40:18Z

distribution/docker/Dockerfile

+
+COPY . /src
+WORKDIR /src
+RUN mvn install -ff -DskipTests -Dforbiddenapis.skip=true -Pdist -Pbundle-contrib-exts


(minor)

You can use cache mounts to make this go faster.

Something like

RUN --mount=type=cache,id=druid-m2-repo,target=/root/.m2/repository mvn install -ff -DskipTests -Dforbiddenapis.skip=true -Pdist -Pbundle-contrib-exts -f /src/pom.xml

I'd prefer not to do this. Its extremely new (released in Docker 18.09) so this will cause trouble in e.g. CI. Also it appears to not universally function with private registry:
moby/moby#32507

distribution/docker/README.md

distribution/docker/druid.sh

distribution/docker/Dockerfile

donbowman · 2019-01-23T23:07:10Z

This is looking pretty good @donbowman

Would you mind posting in the dev list a question about if scripts that download GPL resources are ok in the repository?

Done. I'll watch for responses.

- Remove server/target in favour of target for dockerignore - Remove cd from wget line in favour of -O - Add full path in sha256sums given above wget - Remove 'cd' in .sh script

donbowman · 2019-01-23T23:48:21Z

OK above are all addressed:

posted to ML about script which fetches a GPL file
I don't recommend moving to buildkit requirement given its not avail in CI and has issues with private repo

drcrallen · 2019-01-24T02:28:15Z

I really like this as a way to get experimental clusters up quickly. I would think that the course of adoption would go something like:

Use this docker build method to get an example image built
Run a docker-compose or helm chart or similar to spin up a quick cluster (maybe we can have one in a future pr? proof-of-concept Superset + Druid in a single command would rock). If it were a Mesos framework it could be a full Apache stack :)
Tinker around with taking in some toy data
Show your boss how awesome the tech is
Get a team together to productionize the build / deploy / log / monitor the stuff in whatever way your company does such things.
Profit from awesome interactive analytics
Hire more developers
Get them to be official ASF contributors

This is a great building block for such a funnel.

donbowman · 2019-01-24T02:49:33Z

I really like this as a way to get experimental clusters up quickly. I would think that the course of adoption would go something like:

Use this docker build method to get an example image built

Run a docker-compose or helm chart or similar to spin up a quick cluster (maybe we can have one in a future pr? proof-of-concept Superset + Druid in a single command would rock). If it were a Mesos framework it could a full Apache stack :)

Tinker around with taking in some toy data

Show your boss how awesome the tech is

Get a team together to productionize the build / deploy / log / monitor the stuff in whatever way your company does such things.

Profit from awesome interactive analytics

Hire more developers

Get them to be official ASF contributors

This is a great building block for such a funnel.

I have the docker-compose setup w/ sample data as the next PR. I can easily add superset to it given i use it too.
6,7,8 is up to you :)

maver1ck · 2019-01-25T08:00:36Z

Ad 2)
When this image would be ready I'll move my druid Chart to it. helm/charts#9314

Some environments (e.g. Kubernetes Deployments) don't resolve hostname to IP.

The 32-bit uclibc busybox does not support 64-bit inodes (see GoogleContainerTools/distroless#225) Signed-off-by: Don Bowman <don@agilicus.com>

Signed-off-by: Don Bowman <don@agilicus.com>

donbowman · 2019-02-03T21:18:08Z

there have been no comments on the mailing list.

donbowman · 2019-02-06T16:45:07Z

I'm not sure what else is needed of me here.
No comments received.
I've made the requested changes.

distribution/docker/Dockerfile

drcrallen · 2019-02-06T22:30:02Z

for posterity there is a discussion on the incubator list about docker releases (only tangentially related)

http://mail-archives.apache.org/mod_mbox/incubator-general/201902.mbox/%3CCAGaRif1NXV4ZmakmR3bhGS-UvkOf%3DmJJg3HOKCTEKtaYBW3NDQ%40mail.gmail.com%3E

and the discussion on the druid list seems to have no specific objections and pretty good counter examples:

https://lists.apache.org/thread.html/ed1483e138685207c45a65b22ecc5e83a0236edd285aeb393318b052@%3Cdev.druid.apache.org%3E

drcrallen · 2019-02-06T22:32:16Z

team city issues look unrelated

donbowman · 2019-02-07T19:12:49Z

I have added a sample docker-compose and environment file. This allows ~30s time to get a new local environment up.

This completes the dev, will address any issues.

Signed-off-by: Don Bowman <don@agilicus.com>

This works around issue #3770 #3770 Signed-off-by: Don Bowman <don@agilicus.com>

Signed-off-by: Don Bowman <don@agilicus.com>

maver1ck · 2019-02-08T13:44:10Z

Are we planning to automatically build those images and push to dockerhub ?

donbowman · 2019-02-08T15:14:06Z

@maver1ck see discussion on dev@druid.apache.org.

My vote is obviously yes.

rae89 · 2019-03-25T23:10:30Z

I am having issues building the docker image using the command in the docker readme: docker build -t druid:0.14.0 -f distribution/docker/Dockerfile .

Here is the build failure output:

[INFO] druid-time-min-max ................................. SUCCESS [ 2.133 s]
[INFO] druid-google-extensions ............................ SUCCESS [ 6.508 s]
[INFO] druid-virtual-columns .............................. SUCCESS [ 2.119 s]
[INFO] druid-thrift-extensions ............................ SUCCESS [ 9.566 s]
[INFO] ambari-metrics-emitter ............................. FAILURE [ 2.370 s]
[INFO] sqlserver-metadata-storage ......................... SKIPPED
[INFO] kafka-emitter ...................................... SKIPPED
[INFO] druid-redis-cache .................................. SKIPPED
[INFO] druid-opentsdb-emitter ............................. SKIPPED
[INFO] materialized-view-maintenance ...................... SKIPPED
[INFO] materialized-view-selection ........................ SKIPPED
[INFO] druid-momentsketch ................................. SKIPPED
[INFO] distribution ....................................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 10:02 min
[INFO] Finished at: 2019-03-25T23:08:02Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project ambari-metrics-emitter: Could not resolve dependencies for project org.apache.druid.extensions.contrib:ambari-metrics-emitter:jar:0.15.0-incubating-SNAPSHOT: Could not transfer artifact org.apache.ambari:ambari-metrics-common:jar:2.4.1.0.22 from/to hortonworks (http://repo.hortonworks.com/content/repositories/releases): Failed to transfer file http://repo.hortonworks.com/content/repositories/releases/org/apache/ambari/ambari-metrics-common/2.4.1.0.22/ambari-metrics-common-2.4.1.0.22.jar with status code 503 -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :ambari-metrics-emitter
The command '/bin/sh -c mvn install -ff -DskipTests -Dforbiddenapis.skip=true -Pdist -Pbundle-contrib-exts' returned a non-zero code: 1

donbowman · 2019-03-26T02:30:38Z

It does fetch the files from your local dir (e.g. whatever you have checked out). is it possible there was an error there? I just tried from the RC tag: don@cube[gke-corp]:druid$ git checkout -b druid-0.14.0-incubating-rc1 Switched to a new branch 'druid-0.14.0-incubating-rc1' don@cube[gke-corp]:druid$ docker build -t druid:0.14.0 -f distribution/docker/Dockerfile . ... [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary for Druid 0.15.0-incubating-SNAPSHOT: [INFO] [INFO] Druid .............................................. SUCCESS [ 33.112 s] [INFO] druid-core ......................................... SUCCESS [01:24 min] [INFO] druid-hll .......................................... SUCCESS [ 4.200 s] [INFO] extendedset ........................................ SUCCESS [ 2.976 s] [INFO] druid-processing ................................... SUCCESS [ 21.345 s] [INFO] druid-aws-common ................................... SUCCESS [ 4.630 s] [INFO] druid-console ...................................... SUCCESS [ 24.877 s] [INFO] druid-server ....................................... SUCCESS [ 26.654 s] [INFO] druid-examples ..................................... SUCCESS [ 5.936 s] [INFO] druid-indexing-hadoop .............................. SUCCESS [ 29.969 s] [INFO] druid-indexing-service ............................. SUCCESS [ 7.512 s] [INFO] druid-sql .......................................... SUCCESS [ 9.218 s] [INFO] druid-services ..................................... SUCCESS [ 11.430 s] [INFO] druid-s3-extensions ................................ SUCCESS [ 2.291 s] [INFO] druid-datasketches ................................. SUCCESS [ 3.742 s] [INFO] druid-histogram .................................... SUCCESS [ 2.981 s] [INFO] mysql-metadata-storage ............................. SUCCESS [ 2.301 s] [INFO] druid-kafka-indexing-service ....................... SUCCESS [ 6.335 s] [INFO] druid-basic-security ............................... SUCCESS [ 2.670 s] [INFO] simple-client-sslcontext ........................... SUCCESS [ 1.788 s] [INFO] druid-integration-tests ............................ SUCCESS [ 5.130 s] [INFO] druid-benchmarks ................................... SUCCESS [ 18.045 s] [INFO] druid-avro-extensions .............................. SUCCESS [ 24.472 s] [INFO] druid-bloom-filter ................................. SUCCESS [ 2.969 s] [INFO] druid-kerberos ..................................... SUCCESS [ 2.383 s] [INFO] druid-hdfs-storage ................................. SUCCESS [ 3.115 s] [INFO] druid-stats ........................................ SUCCESS [ 1.587 s] [INFO] druid-kafka-eight .................................. SUCCESS [ 3.535 s] [INFO] druid-lookups-cached-global ........................ SUCCESS [ 4.772 s] [INFO] druid-kafka-extraction-namespace ................... SUCCESS [ 2.186 s] [INFO] druid-kinesis-indexing-service ..................... SUCCESS [ 5.159 s] [INFO] druid-parquet-extensions ........................... SUCCESS [ 4.767 s] [INFO] postgresql-metadata-storage ........................ SUCCESS [ 2.715 s] [INFO] druid-protobuf-extensions .......................... SUCCESS [ 4.891 s] [INFO] druid-lookups-cached-single ........................ SUCCESS [ 2.436 s] [INFO] druid-influx-extensions ............................ SUCCESS [ 1.388 s] [INFO] druid-azure-extensions ............................. SUCCESS [ 1.911 s] [INFO] druid-cassandra-storage ............................ SUCCESS [ 6.038 s] [INFO] druid-rocketmq ..................................... SUCCESS [ 2.398 s] [INFO] druid-cloudfiles-extensions ........................ SUCCESS [ 11.416 s] [INFO] graphite-emitter ................................... SUCCESS [ 2.465 s] [INFO] druid-kafka-eight-simple-consumer .................. SUCCESS [ 1.497 s] [INFO] druid-rabbitmq ..................................... SUCCESS [ 1.860 s] [INFO] druid-distinctcount ................................ SUCCESS [ 1.428 s] [INFO] statsd-emitter ..................................... SUCCESS [ 4.146 s] [INFO] druid-orc-extensions ............................... SUCCESS [ 7.612 s] [INFO] druid-time-min-max ................................. SUCCESS [ 1.394 s] [INFO] druid-google-extensions ............................ SUCCESS [ 3.875 s] [INFO] druid-virtual-columns .............................. SUCCESS [ 1.480 s] [INFO] druid-thrift-extensions ............................ SUCCESS [ 7.749 s] [INFO] ambari-metrics-emitter ............................. SUCCESS [ 3.212 s] [INFO] sqlserver-metadata-storage ......................... SUCCESS [ 1.762 s] [INFO] kafka-emitter ...................................... SUCCESS [ 2.080 s] [INFO] druid-redis-cache .................................. SUCCESS [ 3.056 s] [INFO] druid-opentsdb-emitter ............................. SUCCESS [ 1.802 s] [INFO] materialized-view-maintenance ...................... SUCCESS [ 2.006 s] [INFO] materialized-view-selection ........................ SUCCESS [ 2.072 s] [INFO] druid-momentsketch ................................. SUCCESS [ 1.911 s] [INFO] distribution ....................................... SUCCESS [01:22 min] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 08:54 min [INFO] Finished at: 2019-03-26T02:28:23Z [INFO] ------------------------------------------------------------------------ Removing intermediate container e9517d295277 ...

…

On Mon, 25 Mar 2019 at 19:10, Robert Espinoza ***@***.***> wrote: I am having issues building the docker image using the command in the docker readme: docker build -t druid:0.14.0 -f distribution/docker/Dockerfile . Here is the build failure output: [INFO] druid-time-min-max ................................. SUCCESS [ 2.133 s] [INFO] druid-google-extensions ............................ SUCCESS [ 6.508 s] [INFO] druid-virtual-columns .............................. SUCCESS [ 2.119 s] [INFO] druid-thrift-extensions ............................ SUCCESS [ 9.566 s] [INFO] ambari-metrics-emitter ............................. FAILURE [ 2.370 s] [INFO] sqlserver-metadata-storage ......................... SKIPPED [INFO] kafka-emitter ...................................... SKIPPED [INFO] druid-redis-cache .................................. SKIPPED [INFO] druid-opentsdb-emitter ............................. SKIPPED [INFO] materialized-view-maintenance ...................... SKIPPED [INFO] materialized-view-selection ........................ SKIPPED [INFO] druid-momentsketch ................................. SKIPPED [INFO] distribution ....................................... SKIPPED [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 10:02 min [INFO] Finished at: 2019-03-25T23:08:02Z [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal on project ambari-metrics-emitter: Could not resolve dependencies for project org.apache.druid.extensions.contrib:ambari-metrics-emitter:jar:0.15.0-incubating-SNAPSHOT: Could not transfer artifact org.apache.ambari:ambari-metrics-common:jar:2.4.1.0.22 from/to hortonworks ( http://repo.hortonworks.com/content/repositories/releases): Failed to transfer file http://repo.hortonworks.com/content/repositories/releases/org/apache/ambari/ambari-metrics-common/2.4.1.0.22/ambari-metrics-common-2.4.1.0.22.jar with status code 503 -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn -rf :ambari-metrics-emitter The command '/bin/sh -c mvn install -ff -DskipTests -Dforbiddenapis.skip=true -Pdist -Pbundle-contrib-exts' returned a non-zero code: 1 — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#6896 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AE5Ok9dnuH44NVNsT7npKQdpuFhd_3xEks5vaVd3gaJpZM4aLkB1> .

rae89 · 2019-03-26T22:06:56Z

I still get the error

[ERROR] Failed to execute goal on project ambari-metrics-emitter: Could not resolve dependencies for project org.apache.druid.extensions.contrib:ambari-metrics-emitter:jar:0.15.0-incubating-SNAPSHOT: Could not transfer artifact org.apache.ambari:ambari-metrics-common:jar:2.4.1.0.22 from/to hortonworks (http://repo.hortonworks.com/content/repositories/releases): Failed to transfer file http://repo.hortonworks.com/content/repositories/releases/org/apache/ambari/ambari-metrics-common/2.4.1.0.22/ambari-metrics-common-2.4.1.0.22.jar with status code 503

I also posted on here since it might be related that closed issue:
#4537

donbowman · 2019-03-28T01:01:42Z

I still get the error

[ERROR] Failed to execute goal on project ambari-metrics-emitter: Could not resolve dependencies for project org.apache.druid.extensions.contrib:ambari-metrics-emitter:jar:0.15.0-incubating-SNAPSHOT: Could not transfer artifact org.apache.ambari:ambari-metrics-common:jar:2.4.1.0.22 from/to hortonworks (http://repo.hortonworks.com/content/repositories/releases): Failed to transfer file http://repo.hortonworks.com/content/repositories/releases/org/apache/ambari/ambari-metrics-common/2.4.1.0.22/ambari-metrics-common-2.4.1.0.22.jar with status code 503

I also posted on here since it might be related that closed issue:
#4537

is it possible you have a proxy or firewall in the path?
I can download the referenced file w/o the 503 you get.
maybe a cache?

rae89 · 2019-03-28T16:43:46Z

After finally getting it to build and run, I was wondering how one goes about loading the wikipedia tutorial data described in the loading a file tutorial from the druid docs? http://druid.io/docs/latest/tutorials/tutorial-batch.html

The overlord container does not seem to have bash and python installed to be able to run the script bin/post-index-task.

Or am I missing something here?

donbowman · 2019-03-28T16:46:31Z

you run those from your host, outside the container, pointing @ the ports you exposed.

donbowman force-pushed the Agilicus:add-container-build branch 2 times, most recently from e7d091b to 06994b2 Jan 21, 2019

donbowman force-pushed the Agilicus:add-container-build branch from 06994b2 to a7015f7 Jan 22, 2019

drcrallen requested changes Jan 23, 2019

View changes

drcrallen reviewed Jan 23, 2019

View changes

distribution/docker/Dockerfile Outdated Show resolved Hide resolved

donbowman added 3 commits Jan 23, 2019

Add Apache license header

Verified

This commit was signed with a verified signature.

donbowman Don Bowman

GPG key ID: D21D0E75160D9894 Learn about signing commits

Loading status checks…

a15ddf6

Add comments at top of script per PR comments

Verified

This commit was signed with a verified signature.

donbowman Don Bowman

GPG key ID: D21D0E75160D9894 Learn about signing commits

Loading status checks…

b5afa65

Revert BUILDKIT. Not available everywhere.

Verified

This commit was signed with a verified signature.

donbowman Don Bowman

GPG key ID: D21D0E75160D9894 Learn about signing commits

Loading status checks…

cba8786

donbowman mentioned this pull request Jan 24, 2019

[incubator/druid] New Druid Chart helm/charts#9314

Merged

2 of 2 tasks complete

donbowman force-pushed the Agilicus:add-container-build branch from f4b5daa to 729dbe7 Jan 25, 2019

donbowman added 2 commits Jan 28, 2019

Override service-specific configuration

Verified

This commit was signed with a verified signature.

donbowman Don Bowman

GPG key ID: D21D0E75160D9894 Learn about signing commits

Loading status checks…

de2328a

Signed-off-by: Don Bowman <don@agilicus.com>

dylwylie reviewed Feb 6, 2019

View changes

distribution/docker/Dockerfile Outdated Show resolved Hide resolved

distribution/docker/Dockerfile Show resolved Hide resolved

drcrallen approved these changes Feb 6, 2019

View changes

drcrallen added Ease of Use Area - Operations labels Feb 7, 2019

Replace MAINTAINER w/ LABEL

Verified

This commit was signed with a verified signature.

donbowman Don Bowman

GPG key ID: D21D0E75160D9894 Learn about signing commits

83c5ded

Signed-off-by: Don Bowman <don@agilicus.com>

donbowman added 2 commits Feb 7, 2019

Add docker-compose and sample environment

Verified

This commit was signed with a verified signature.

donbowman Don Bowman

GPG key ID: D21D0E75160D9894 Learn about signing commits

Loading status checks…

7106b19

Signed-off-by: Don Bowman <don@agilicus.com>

donbowman closed this Feb 7, 2019

donbowman reopened this Feb 7, 2019

donbowman closed this Feb 7, 2019

donbowman reopened this Feb 7, 2019

dylwylie approved these changes Feb 8, 2019

View changes

dylwylie merged commit b3dcbe7 into apache:master Feb 8, 2019
2 checks passed

2 checks passed

Inspections: pull requests (Druid) TeamCity build finished
Details

continuous-integration/travis-ci/pr The Travis CI build passed
Details

donbowman deleted the Agilicus:add-container-build branch Mar 28, 2019

clintropolis mentioned this pull request Apr 5, 2019

change Dockerfile maintainers email to dev mailing list #7417

Merged

jihoonson added this to the 0.15.0 milestone May 15, 2019

apache / druid

Add docker container for druid #6896

Add docker container for druid #6896

donbowman commented Jan 21, 2019

drcrallen left a comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

donbowman commented Jan 23, 2019

donbowman commented Jan 23, 2019

drcrallen commented Jan 24, 2019 •

edited

donbowman commented Jan 24, 2019

maver1ck commented Jan 25, 2019

donbowman commented Feb 3, 2019

donbowman commented Feb 6, 2019

drcrallen commented Feb 6, 2019

drcrallen commented Feb 6, 2019

donbowman commented Feb 7, 2019 •

edited

maver1ck commented Feb 8, 2019

donbowman commented Feb 8, 2019

rae89 commented Mar 25, 2019

donbowman commented Mar 26, 2019

rae89 commented Mar 26, 2019

donbowman commented Mar 28, 2019

rae89 commented Mar 28, 2019

donbowman commented Mar 28, 2019

apache / druid

Join GitHub today

GitHub is where the world builds software

Add docker container for druid #6896

Add docker container for druid #6896

Conversation

donbowman commented Jan 21, 2019

drcrallen left a comment

This comment has been minimized.

drcrallen Jan 23, 2019 Contributor

This comment has been minimized.

drcrallen Jan 23, 2019 Contributor

This comment has been minimized.

donbowman Jan 24, 2019 Author Contributor

This comment has been minimized.

drcrallen Jan 23, 2019 Contributor

This comment has been minimized.

donbowman Jan 23, 2019 Author Contributor

donbowman commented Jan 23, 2019

donbowman commented Jan 23, 2019

drcrallen commented Jan 24, 2019 • edited

donbowman commented Jan 24, 2019

maver1ck commented Jan 25, 2019

donbowman commented Feb 3, 2019

donbowman commented Feb 6, 2019

drcrallen commented Feb 6, 2019

drcrallen commented Feb 6, 2019

donbowman commented Feb 7, 2019 • edited

maver1ck commented Feb 8, 2019

donbowman commented Feb 8, 2019

rae89 commented Mar 25, 2019

donbowman commented Mar 26, 2019

rae89 commented Mar 26, 2019

donbowman commented Mar 28, 2019

rae89 commented Mar 28, 2019

donbowman commented Mar 28, 2019

Essential cookies

Always active

Analytics cookies

drcrallen Jan 23, 2019
Contributor

drcrallen Jan 23, 2019
Contributor

donbowman Jan 24, 2019
Author Contributor

drcrallen Jan 23, 2019
Contributor

donbowman Jan 23, 2019
Author Contributor

drcrallen commented Jan 24, 2019 •

edited

donbowman commented Feb 7, 2019 •

edited