Spark in me - Internet, data science, math, deep learning, philo

snakers4 @ telegram, 1252 members, 1404 posts since 2016

All this - lost like tears in rain.

Internet, data science, math, deep learning, philosophy. No bs.

Our website
- spark-in.me
Our chat
- goo.gl/WRm93d
DS courses review
- goo.gl/5VGU5A
- goo.gl/YzVUKf

Posts by tag «deep_learning»:

snakers4 (Alexander), April 18, 13:37

Nice ideas about unit testing ML code

medium.com/@keeper6928/how-to-unit-test-machine-learning-code-57cf6fd81765

#deep_learning

How to unit test machine learning code.

Note: The popularity of this post has inspired me to write a machine learning test library. Go check it out!


snakers4 (Alexander), April 16, 10:17

A draft of the article about DS Bowl 2018 on Kaggle.

This time this was a lottery.

Good that I did not really spend much time, but this time I learned a lot about watershed and some other instance segmentation methods!

An article is accompanied by a dockerized PyTorch code release on GitHub:

- spark-in.me/post/playing-with-dwt-and-ds-bowl-2018

- github.com/snakers4/ds_bowl_2018

This is a beta, you are welcome to comment and respond.

Kudos!

#data_science

#deep_learning

#instance_se

Applying Deep Watershed Transform to Kaggle Data Science Bowl 2018 (dockerized solution)

In this article I will describe my solution to the DS Bowl 2018 and why it was a lottery and post a link to my dockerized solution Статьи автора - http://spark-in.me/author/snakers41 Блог - http://spark-in.me


snakers4 (Alexander), April 15, 09:49

A DISCIPLINED APPROACH TO NEURAL NETWORK HYPER-PARAMETERS: PART 1 – LEARNING RATE, BATCH SIZE, MOMENTUM, AND WEIGHT DECAY

(0) arxiv.org/abs/1803.09820, Leslie N. Smith US Naval Research Laboratory

(1) Will serve as a good intuition starter if you have little experience (!)

(2) Some nice ideas:

- The test/validation loss is a good indicator of the network’s convergence - especially in early epochs

- The amount of regularization must be balanced for each dataset and architecture

- The practitioner’s goal is obtaining the highest performance while minimizing the needed computational time

(smaller batch - less stability and faster convergence)

- Optimal momentum value(s) will improve network training

(3) The author does not study the difference between SGD and Adam in depth =( Adam kind of solves much of his pains

(4) In my practice the following approach works best:

- Aggressive training with Adam to find the optimal LR

- Apply various LR decay regimes to determine the optimal

- Use low LR or CLR in the end to converge to a lower value (possible overfitting)

- Test on test / delayed test end-to-end

- In my experience - a strong model with good params will start with test/val set loss much lower / target metric much higher than on the train set

- In some applications if your CNN is memory intesive - you just opt for the largest batch possible (usually >6-8 works)

- Also there is no mention of augmentations - they usually help reduce overfitting much better than hyper parameters

#deep_learning

Nice read about systemctl

www.digitalocean.com/community/tutorials/how-to-use-systemctl-to-manage-systemd-services-and-units

#linux

How To Use Systemctl to Manage Systemd Services and Units | DigitalOcean

Systemd is an init system and system manager that is widely becoming the new standard for Linux machines. While there is considerable controversy as to whether systemd is an improvement over the init systems it is replacing, the majority of distributi


snakers4 (Alexander), April 15, 08:06

2018 DS/ML digest 8

As usual my short bi-weekly (or less) digest of everything that passed my BS detector

Market / blog posts

(0) Fast.ai about the importance of accessibility in ML - www.fast.ai/2018/04/10/stanford-salon/

(1) Some interesting news about market, mostly self-driving cars (the rest is crap) - goo.gl/VKLf48

(2) US$600m investment into Chinese face recognition - goo.gl/U4k2Mg

Libraries / frameworks / tools

(0) New 5 point face detector in Dlib for face alignment task - goo.gl/T73nHV

(1) Finally a more proper comparsion of XGB / LightGBM / CatBoost - goo.gl/AcszWZ (also see my thoughts here snakers41.spark-in.me/1840)

(3) CNNs on FPGAs by ZFTurbo

-- www.youtube.com/watch?v=Lhnf596o0cc

-- github.com/ZFTurbo/Verilog-Generator-of-Neural-Net-Digit-Detector-for-FPGA

(4) Data version control - looks cool

-- dataversioncontrol.com

-- goo.gl/kx6Qdf

-- but I will not use it - becasuse proper logging and treating data as immutable solves the issue

-- looks like over-engineering for the sake of overengineering (unless you create 100500 datasets per day)

Visualizations

(0) TF Playground to seed how simplest CNNs work - goo.gl/cu7zTm

Applications

(0) Looks like GAN + ResNet + Unet + content loss - can easily solve simpler tasks like deblurring goo.gl/aviuNm

(1) You can apply dilated convolutions to NLP tasks - habrahabr.ru/company/ods/blog/353060/

(2) High level overview of face detection in ok.ru - goo.gl/fDUXa2

(3) Alternatives to DWT and Mask-RCNN / RetinaNet? medium.com/@barvinograd1/instance-embedding-instance-segmentation-without-proposals-31946a7c53e1

- Has anybody tried anything here?

Papers

(0) A more disciplined approach to training CNNs - arxiv.org/abs/1803.09820 (LR regime, hyper param fitting etc)

(1) GANS for iamge compression - arxiv.org/pdf/1804.02958.pdf

(2) Paper reviews from ODS - mostly moonshots, but some are interesting

-- habrahabr.ru/company/ods/blog/352508/

-- habrahabr.ru/company/ods/blog/352518/

(3) SqueezeNext - the new SqueezeNet - arxiv.org/abs/1803.10615

#digest

#data_science

#deep_learning

snakers4 (Alexander), April 12, 08:08

DS Bowl 2018 stage 2 data was released.

It has completely different distribution from stage 1 data.

How do you like them, apples?

Looks like Kaggle admins really have no idea about dataset curation, or all of this is mean to misguide manual annotators.

Anyway - looks like random bs.

#data_science

#deep_learning

snakers4 (Alexander), April 10, 09:01

A bit more on semantic segmentation, now 3D

{V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation

}

--> Link / authors arxiv.org/abs/1606.04797, Fausto Milletari / Nassir Navab / Seyed-Ahmad Ahmadi

--> Essence:

(0) Essentially applies UNet to 3D with a custom DICE based loss

(1) Architecture - goo.gl/Yn2BGb - basically UNet with 3D convolutions. Upsampling / downsampling - goo.gl/VtXrXy

(2) PReLu (no ablation test)

(3) Receptive fields of layers - goo.gl/FGwDCF

(4) 3D DICE loss - goo.gl/SqrK93 (wo BCE?)

--> The paper does not use all the juice possible - hacky transfer learning (obvious idea - just stacking Imagenet filters), CLR, LinkNet architectures, etc

--> Looks like a good baseline / reference

{An application of cascaded 3D fully convolutional networks for medical image segmentation

}

--> arxiv.org/abs/1803.05431, a group of Japanese researchers

--> Essence:

(0) 2 stage 3D UNet, ablation test against 2D FCNs

(1) Loss - 3D cross-entropy

(2) Transfer learning - it works for other datasets, give a mild boost (1-3 %)

(3) 80-90% DICE, varies by organ

(4) weights downloadable github.com/holgerroth/3Dunet_abdomen_cascade (Caffe...)

--> Essentially a 2 stage process is dictated by memory considerations:

(0) Pipeline goo.gl/wZwF3X

In the long run transfer learning may rule, but here legal limitations may slow down this process.

#deep_learning

#medical_imaging

snakers4 (Alexander), April 08, 13:48

As you may know (for newer people on the channel), sometimes we publish small articles on the website.

This time it covers a recent Power Laws challenge on DrivenData, which at first seemed legit and cool, but in the end turned back into a pumpkin.

Here is an article:

- spark-in.me/post/playing-with-electricity

#data_science

#time_series

#deep_learning

Playing with electricity - forecasting 5000 time series

In this article I share our experience participating in a recent time series challenge on Drivendata and my personal ideas about ML competitions Статьи автора - http://spark-in.me/author/snakers41 Блог - http://spark-in.me


snakers4 (Alexander), April 07, 11:14

The trend for smaller / inadequate prizes and weird datasets continues:

- This years' DS Bowl on Kaggle features a small public train dataset (650 images) vs. much larger delayed validation dataset (3000 images). Cheap annotation, anyone? =) Also remarkably, for a somewhat difficult task (instance segmentation) - the prize is much lower than the last year;

- New autonomous driving contest on Kaggle, as well as other CVPR competitions - features extremely large datasets, extremely low prizes (US$1-2k), and no travel costs to CVPR covered. Ofc you can win and fly there, but this will not even cover your GPU costs;

- The recent xnView challenge I really wanted to participate - requires a US Tax number to be eligible for prizes. Of course they do not know about double taxation treaties and WEP-8 tax exemptions;

Alas =(

#deep_learning

snakers4 (Alexander), April 05, 06:19

For some reason when you use PyTorch multi-thread data loaders, it stalls if you use OpenCV and not set

cv2.setNumThreads(0)

Nice to know this.

#deep_learning

snakers4 (Alexander), April 03, 07:53

Handy opencv snippet to transform grayscale mask with labels (i.e. 0 for background, 1,2,3 etc) into a colourful map

y_pred_coloured = cv2.applyColorMap((y_pred / y_pred.max() * 255).astype('uint8'), cv2.COLORMAP_JET)

#deep_learning

snakers4 (Alexander), March 30, 04:35

New TF release (if you are into it, I personally use TensorBoard)

- github.com/tensorflow/tensorflow/releases/tag/v1.7.0

What is interesting

TensorBoard Debugger Plugin, the graphical user interface (GUI) of TensorFlow Debugger (tfdbg), is now in alpha.

#deep_learning

tensorflow/tensorflow

tensorflow - Computation using data flow graphs for scalable machine learning


A new post by Fchollet

medium.com/@francois.chollet/what-worries-me-about-ai-ed9df072b704

#deep_learning

What worries me about AI

Disclaimer: These are my own personal views. I do not speak for my employer. If you quote this article, please have the honesty to present…


snakers4 (Alexander), March 20, 13:40

A video about realistic state of chat-bots (RU)

www.facebook.com/deepmipt/videos/vb.576961855845861/868578240017553/?type=2&theater

#nlp

#data_science

#deep_learning

Neural Networks and Deep Learning lab at MIPT

При звонке в сервис или банк диалог с поддержкой строится по одному и тому же сценарию с небольшими вариациями. Отвечать «по бумажке» на вопросы может и не человек, а чат-бот. О том, как нейронные...


snakers4 (Alexander), March 14, 09:53

A nice thread about ML reproducibility

www.reddit.com/r/MachineLearning/comments/845uji/d_are_the_hyperrealistic_results_of_tacotron2_and/

#deep_learning

[D] Are the hyper-realistic results of... • r/MachineLearning

**Tacotron-2:** https://research.googleblog.com/2017/12/tacotron-2-generating-human-like-speech.html **Wavenet:**...


snakers4 (Alexander), March 10, 13:59

Interesting / noteworthy semseg papers

In practice - UNet and LinkNet are best and simple solutions.

Rarely people report that something like Tiramisu works properly.

Though I saw once in last Konika competition - a good solution based on DenseNet + Standard decoder.

So I decided to read some of the newer and older Semseg papers.

Classic papers

UNet,LinkNet - nuff said

(0) Links

- UNet - arxiv.org/abs/1505.04597

- LinkNet - arxiv.org/abs/1707.03718

Older, overlooked, but interesting papers

SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation

One of original papers before UNet

(0) arxiv.org/abs/1511.00561

(1) Basically UNet w/o skip connections but it stores pooling indices

(1) SegNet uses the max pooling indices to upsample (without learning) the feature map(s) and convolves with a trainable decoder filter bank

ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation

Paszke, Adam / Chaurasia, Abhishek / Kim, Sangpil / Culurciello, Eugenio

(0) Link

- arxiv.org/abs/1606.02147

(1) Key facts

- Up to 18× faster, 75× less FLOPs, 79× less parameters vs SegNet or FCN

- Supposedly runs on NVIDIA Jetson TX1 Embedded Systems

- Essentially a minzture of ResNet and Inception architectures

- Overview of the architecture

-- goo.gl/M6CPEv

-- goo.gl/b5Kb2S

(2) Interesting ideas

- Visual information is highly spatially redundant, and thus can be compressed into a more efficient representation

- Highly assymetric - decoder is much smaller

- Dilated convolutions in the middle => significant accuracy boost

- Dropout > L2

- Pooling operation in parallel with a convolution of stride 2, and concatenate resulting feature maps

Newer papers

xView: Objects in Context in Overhead Imagery - new "Imagenet" for satellite images

(0) Link

- Will be available here xviewdataset.org/#register

(1) Examples

- goo.gl/JKr9wW

- goo.gl/TWRmn2

(2) Stats

- 0.3m ground sample distance

- 60 classes in 7 different parent classes

- 1 million labeled objects covering over 1,400 km2 of the earth’s surface

- classes goo.gl/v9CM5b

(3) Baseline

- Their baseline using SSD has very poor performance ~20% mAP

Rethinking Atrous Convolution for Semantic Image Segmentation

(0) Link

- arxiv.org/abs/1706.05587

- Liang-Chieh Chen George / Papandreou Florian / Schroff Hartwig / Adam

- Google Inc.

(1) Problems to be solved

- Reduced feature resolution

- Objects at multiple scales

(2) Key approaches

- Image pyramid (reportedly works poorly and requires a lot of memory)

- Encoder-decoder

- Spatial pyramid pooling (reportedly works poorly and requires a lot of memory)

(3) Key ideas

- Atrous (dilated) convolution - goo.gl/uSFCv5

- ResNet + Atrous convolutions - goo.gl/pUjUBS

- Atrous Spatial Pyramid Pooling block goo.gl/AiQZC1 - goo.gl/p63qNR

(4) Performance

- As with the latest semseg methods, true performance boost is unclear

- I would argue that such methods may be useful for large objects

#digest

#deep_learning

snakers4 (Alexander), March 09, 05:49

PyTorch caught up with keras

mobile.twitter.com/i/web/status/971863128341323776

#deep_learning

François Chollet

TensorFlow is the platform of choice for deep learning in the research community. These are deep learning framework mentions on arXiv over the past 3 months


snakers4 (Alexander), March 07, 06:03

New articles about picking GPUs for DL

- blog.slavv.com/picking-a-gpu-for-deep-learning-3d4795c273b9

- goo.gl/h6PJqc

Also most likely for 4-GPU set-ups in a case some of them may be required to have water cooling to avoid thermal throttling.

#deep_learning

Picking a GPU for Deep Learning

Buyer’s guide at the beginning of 2018


snakers4 (Alexander), March 07, 05:03

2018 DS/ML digest 6

Visualization

(1) A new amazing post by Google on distil - distill.pub/2018/building-blocks/.

This is really amazing work, but their notebooks tells me that it is a far cry from being able to be utilized by the community - goo.gl/3c1Fza

This is how the CNN sees the image - goo.gl/S4KT5d

Expect this to be packaged as part of Tensorboard in a year or so)

Datasets

(1) New landmark dataset by Google - goo.gl/veSEhg - looks cool, but ...

Prizes in the accompanying Kaggle competitions are laughable goo.gl/EEGDEH goo.gl/JF93Xx

Given that datasets are really huge...~300G

Also also if you win, you will have to buy a ticket to the USA on your money ...

(2) Useful script to download the images goo.gl/JF93Xx

(3) Imagenet for satellite imagery - xviewdataset.org/#register - pre-register

arxiv.org/pdf/1802.07856.pdf paper

(4) CVPR 2018 for satellite imagery - deepglobe.org/challenge.html

Papers / new techniques

(1) Improving RNN performance via auxiliary loss - arxiv.org/pdf/1803.00144.pdf

(2) Satellite imaging for emergencies - arxiv.org/pdf/1803.00397.pdf

(3) Baidu - neural voice cloning - goo.gl/uJe852

Market

(1) Google TPU benchmarks - goo.gl/YKL9yx

As usual such charts do not show consumer hardware.

My guess is that a single 1080Ti may deliver comparable performance (i.e. 30-40% of it) for ~US$700-1000k, i.e. ~150 hours of rent (this is ~ 1 week!)

Miners say that 1080Ti can work 1-2 years non-stop

(2) MIT and SenseTime announce effort to advance artificial intelligence research goo.gl/MXB3V9

(3) Google released its ML course - goo.gl/jnVyNF - but generally it is a big TF ad ... Andrew Ng is better for grasping concepts

Internet

(1) Interesting thing - all ISPs have some preferential agreements between each other - goo.gl/sEvZMN

#digest

#data_science

#deep_learning

The Building Blocks of Interpretability

Interpretability techniques are normally studied in isolation. We explore the powerful interfaces that arise when you combine them -- and the rich structure of this combinatorial space.


snakers4 (Alexander), March 06, 14:34

I have seen questions on forums - how to add Keras-like progress bar to PyTorch for simple models?

The answer is to use tqdm and this property

- goo.gl/cG6Ug8

This example is also great

from tqdm import trange

from random import random, randint

from time import sleep

t = trange(100)

for i in t:

# Description will be displayed on the left

t.set_description('GEN %i' % i)

# Postfix will be displayed on the right, and will format automatically

# based on argument's datatype

t.set_postfix(loss=random(), gen=randint(1,999), str='h', lst=[1, 2])

sleep(0.1)

#deep_learning

#pytorch

Can I add message to the tqdm progressbar?

When using the tqdm progress bar: can I add a message to the same line as the progress bar in a loop? I tried using the "tqdm.write" option, but it adds a new line on every write. I would like each


snakers4 (Alexander), February 23, 06:30

Was looking for CLAHE abstraction for my image pre-processing pipeline and found one in the Internet

class CLAHE:

def __init__(self, clipLimit=2.0, tileGridSize=(8, 8)):

self.clipLimit = clipLimit

self.tileGridSize = tileGridSize

def __call__(self, im):

img_yuv = cv2.cvtColor(im, cv2.COLOR_BGR2YUV)

clahe = cv2.createCLAHE(clipLimit=self.clipLimit, tileGridSize=self.tileGridSize)

img_yuv[:, :, 0] = clahe.apply(img_yuv[:, :, 0])

img_output = cv2.cvtColor(img_yuv, cv2.COLOR_YUV2BGR)

return img_output

#deep_learning

snakers4 (Alexander), February 20, 04:16

So a couple of things - goo.gl/CBk7Um

- Movidius USB stick is enough to launch real-time object detection which is interesting to know

- It has shitty driver and library support (Caffe was mentioned)

- Installing everything is FAR from trivial (no idea why virtual box was used, but whatever)

- This guide uses Virtual box instead of Docker which says much

Also PyImageSearch is a sellout - he most likely has advertiser-friendly featured content in this post, looks like the Movidius stick topcoder event did not gain enough traction...

So - use Nvidia Jetsons for embedded solutions and do not bother with this. But it's good that new products emerge.

#deep_learning

Real-time object detection on the Raspberry Pi with the Movidius NCS - PyImageSearch

In this tutorial I'll demonstrate how you an achieve real-time object detection on the Raspberry Pi using deep learning and Intel's Movidius NCS.


snakers4 (Alexander), February 19, 10:25

One more article about usual suspects when your CNN fails to train

- blog.slavv.com/37-reasons-why-your-neural-network-is-not-working-4020854bd607

#deep_learning

37 Reasons why your Neural Network is not working

The network had been training for the last 12 hours. It all looked good: the gradients were flowing and the loss was decreasing. But then…


snakers4 (Alexander), February 18, 08:05

Even though I am preparing a large release on GAN application on real example, I just could not help sharing these 2 links.

They are just an absolute of perfection for GANs on PyTroch

- github.com/martinarjovsky/WassersteinGAN

- github.com/soumith/ganhacks

Also this is the most idiomatic PyTorch code (Imagenet finetuning) code I have ever seen

- gist.github.com/panovr/2977d9f26866b05583b0c40d88a315bf

So if you are new to PyTorch, then these links will be very useful)

#pytorch

#deep_learning

#gans

martinarjovsky/WassersteinGAN

Contribute to WassersteinGAN development by creating an account on GitHub.


Which of the latest projects did you like the most?

Still waiting for GANs – 17

👍👍👍👍👍👍👍 49%

Satellites! – 9

👍👍👍👍 26%

Nothing / not interested / missed them – 5

👍👍 14%

Jungle! – 3

👍 9%

PM me for other options – 1

▫️ 3%

👥 35 people voted so far.

snakers4 (Alexander), February 14, 11:48

2017 DS/ML digest 4

Applied cool stuff

- How Dropbox build their OCR - via CTC loss - goo.gl/Dumcn9

Fun stuff

- CNN forward pass done in Google Sheets - goo.gl/pyr44P

- New Boston Robotics robot - opens doors now - goo.gl/y6G5bo

- Cool but toothless list of jupyter notebooks with illustrations and models modeldepot.io

- Best CNN filter visualization tool ever - ezyang.github.io/convolution-visualizer/index.html

New directions / moonshots / papers

- IMPALA from Google - DMLab-30, a set of new tasks that span a large variety of challenges in a visually unified environment with a common action space

-- goo.gl/7ASXdk

-- twitter.com/DeepMindAI/status/961283614993539072

- Trade crypto via RL - goo.gl/NmCQSY?

- SparseNets? - arxiv.org/pdf/1801.05895.pdf

- Use Apple watch data to predict diseases arxiv.org/abs/1802.02511?

- Google - Evolution in auto ML kicks in faster than RL - arxiv.org/pdf/1802.01548.pdf

- R-CNN for human pose estimation + dataset

-- Website + video densepose.org

-- Paper arxiv.org/abs/1802.00434

Google's Colaboratory gives free GPUs?

- Old GPUs

- 12 hours limit, but very cool in theory

- habrahabr.ru/post/348058/

- www.kaggle.com/getting-started/47096#post271139

Sick sad world

- China has police Google Glass with face recognition goo.gl/qfNGk7

- Why slack sucks - habrahabr.ru/post/348898/

-- Email + google docs is better for real communication

Market

- Globally there are 22k ML developers goo.gl/1Jpt9P

- One more AI chip moonshot - goo.gl/199f5t

- Google made their TPUs public in beta - US$6 per hour

- CNN performance comparable to human level in dermatology (R-CNN) - goo.gl/gtgXVn

- Deep learning is greedy, brittle, opaque, and shallow goo.gl/7amqxB

- One more medical ML investment - US$25m for cancer - goo.gl/anndPP

#digest

#data_science

#deep_learning

snakers4 (Alexander), February 14, 04:54

Article on SpaceNet Challenge Three in Russian on habrhabr - please support us with your comments / upvotes

- habrahabr.ru/post/349068/

Also if you missed:

- The original article spark-in.me/post/spacenet-three-challenge

- The original code release github.com/snakers4/spacenet-three

... and Jeremy Howard from fast.ai retweeted our solution, lol

- twitter.com/alxndrkalinin/status/961268736077582336

=)

But to give some idea which pain the TopCoder platform induces on the contestants, you can read

- Data Download guide goo.gl/EME8nA

- Final testing guide goo.gl/DCvTNN

- Code release for their verification process

github.com/snakers4/spacenet-three-topcoder

#data_science

#deep_learning

#satelling_imaging

Из спутниковых снимков в графы (cоревнование SpaceNet Road Detector) — попадание топ-10 и код (перевод)

Привет, Хабр! Представляю вам перевод статьи. Это Вегас с предоставленной разметкой, тестовым датасетом и вероятно белые квадраты — это отложенная валидация...


snakers4 (Alexander), February 13, 07:55

Interesting hack from n01z3 from ODS

For getting that extra 1%

Snapshot Ensembles / Multi-checkpoint TTA:

- goo.gl/f5D2ER

- Train CNN with LR decay until convergence, use SGD or Adam

- Use cyclic LR starting to train the network from the best checkpoint, train for several epochs

- Collect checkpoints with the best loss and use them for ensembles / TTA

#deep_learning

Google TPUs are released in beta..US$200 per day?

No thank you! Also looks like only TF is supported so far.

Combined with rumours, sounds impractical.

- goo.gl/UDkE8B

#deep_learning

Cloud TPU machine learning accelerators now available in beta

By John Barrus, Product Manager for Cloud TPUs, Google Cloud and Zak Stone, Product Manager for TensorFlow and Cloud TPUs, Google Brain Team...


snakers4 (Alexander), February 10, 07:13

So we started publishing articles / code / solutions to the recent SpaceNet3 challenge. A Russian article on habrhabr.ru will also be published soon.

- The original article spark-in.me/post/spacenet-three-challenge

- The original code release github.com/snakers4/spacenet-three

... and Jeremy Howard from fast.ai retweeted our solution, lol

- twitter.com/alxndrkalinin/status/961268736077582336

=)

But to give some idea which pain the TopCoder platform induces on the contestants, you can read

- Data Download guide goo.gl/EME8nA

- Final testing guide goo.gl/DCvTNN

- Code release for their verification process

github.com/snakers4/spacenet-three-topcoder

#data_science

#deep_learning

#satelling_imaging

How we participated in SpaceNet three Road Detector challenge

This article tells about our SpaceNet Challenge participation, semantic segmentation in general and transforming masks into graphs Статьи автора - http://spark-in.me/author/snakers41 Блог - http://spark-in.me


snakers4 (Alexander), February 08, 09:13

Fast.ai lesson 11 notes:

- Links

-- Video www.youtube.com/watch?v=bZmJvmxfH6I&feature=youtu.be

-- course.fast.ai/lessons/lesson11.html

- Semantic embeddings + imagenet can be powerful, but not deployable per se

- Training nets on smaller images usually works

- Comparing activation functions goo.gl/7JakeK

- lr annealing goo.gl/MEu1p8

- linear learnable colour swap trick goo.gl/DnCsHw

- adding Batchnorm goo.gl/Bh8Evh

- replacing max-pooling with avg_pooling goo.gl/1nhPvq

- lr vs batch-size goo.gl/TGJjy7

- dealing with noisy labels goo.gl/RhWzov

- FC / max-pooling layer models are better for transfer-learning?

- size vs. flops vs. speed goo.gl/GjfY4p

- cyclical learning rate paper goo.gl/3XigT9

- Some nice intuitions about mean shift clustering

-- spin.atomicobject.com/2015/05/26/mean-shift-clustering/

-- vision.stanford.edu/teaching/cs131_fall1314_nope/lectures/lecture13_kmeans_cs131.pdf

#data_science

#deep_learning

Lesson 11: Cutting Edge Deep Learning for Coders
We’ve covered a lot of different architectures, training algorithms, and all kinds of other CNN tricks during this course—so you might be wondering: what sho...

Meta research on the CNNs

(also this amazing post towardsdatascience.com/neural-network-architectures-156e5bad51ba)

An Analysis of Deep Neural Network Models for Practical Applications

arxiv.org/abs/1605.07678

Key findings:

(1) power consumption is independent of batch size and architecture;

(2) accuracy and inference time are in a hyperbolic relationship;

(3) energy constraint = upper bound on the maximum achievable accuracy and model complexity;

(4) the number of operations is a reliable estimate of the inference time

Charts

- Accuracy and param number - goo.gl/Zh6uGd

- Param efficiency - goo.gl/PrVtd5

Also a summary of architectural patterns

- goo.gl/KuS2ja

======================================

Deep Learning Scaling is Predictable, Empirically

arxiv.org/abs/1712.00409

TLDR

- goo.gl/6hdThJ

Implications

- various empirical learning curves show robust power-law region

- new architectures slightly shift learning curves downwards

- model architecture exploration should be feasible with small training data sets

- it can be difficult to ensure that training data is large enough to see the power-law learning curve region

- irreducible error region

- each new hardware generation with improved FLOP rate can pro- vide a predictable step function improvement in relative DL model accuracy

#data_science

#deep_learning

Neural Network Architectures

Deep neural networks and Deep Learning are powerful and popular algorithms. And a lot of their success lays in the careful design of the…


snakers4 (Alexander), February 07, 14:09

Following our blog post, we also posted a Russian translation of the Jungle competition to habrhabr

- habrahabr.ru/post/348540/

#data_science

#deep_learning

Соревнование Pri-matrix Factorization на DrivenData с 1ТБ данных — как мы заняли 3 место (перевод)

Привет, Хабр! Представляю вашему вниманию перевод статьи "Animal detection in the jungle — 1TB+ of data, 90%+ accuracy and 3rd place in the competition". Или...


snakers4 (Alexander), February 06, 05:23

We are starting to publish our code / solutions / articles from recent competitions (Jungle and SpaceNet three).

This time the code will be more polished / idiomatic, so that you can learn something from it!

Jungle competition

- Finally it was verified that we indeed won the 3rd place)

- drivendata.org/competitions/49/deep-learning-camera-trap-animals/

Blog posts

- spark-in.me/post/jungle-animal-trap-competition-part-one

- spark-in.me/post/jungle-animal-trap-competition-part-one

- An adaptation for habrhabr.ru will be coming soon

Code release and architecture:

- Code github.com/snakers4/jungle-dd-av-sk

- Architecture

-- 1st place (kudos to Dmytro) - simple and nice goo.gl/19S6WJ

-- Ours goo.gl/st2mGS

-- 2nd place - 4-5 levels of stacking goo.gl/wn5vEW

Please comment under posts / share / buy us a coffee!

- Buy a coffee buymeacoff.ee/8oneCIN

- Rate our channel tg://resolve?domain=tchannelsbot&start=snakers4

#data_science

#deep_learning

#competitions

Pri-matrix Factorization

Chimp&See has collected nearly 8,000 hours of footage reflecting chimpanzee habitats from camera traps across Africa. Your challenge is to build a model that identifies the wildlife in these videos.


snakers4 (Alexander), February 05, 14:57

We also managed to get into top-10 in SpaceNet3 Road Detection challenge

- goo.gl/hswjGp

(Final confirmation awaits)

Here is a sneak peak of our solution

- goo.gl/2yAZAE

A blog post + repo will follow

#data_science

#deep_learning

#satellite_imaging

#graphs

Flowchart Maker & Online Diagram Software

draw.io is a free online diagramming application and flowchart maker . You can use it to create UML, entity relationship, org charts, BPMN and BPM, database schema and networks. Also possible are telecommunication network, workflow, flowcharts, maps overlays and GIS, electronic circuit and social network diagrams.