Spark in me - Internet, data science, math, deep learning, philo

snakers4 @ telegram, 1252 members, 1404 posts since 2016

All this - lost like tears in rain.

Internet, data science, math, deep learning, philosophy. No bs.

Our website
Our chat
DS courses review

Posts by tag «deep_learning»:

snakers4 (Alexander), April 18, 13:37

Nice ideas about unit testing ML code


How to unit test machine learning code.

Note: The popularity of this post has inspired me to write a machine learning test library. Go check it out!

snakers4 (Alexander), April 16, 10:17

A draft of the article about DS Bowl 2018 on Kaggle.

This time this was a lottery.

Good that I did not really spend much time, but this time I learned a lot about watershed and some other instance segmentation methods!

An article is accompanied by a dockerized PyTorch code release on GitHub:



This is a beta, you are welcome to comment and respond.





Applying Deep Watershed Transform to Kaggle Data Science Bowl 2018 (dockerized solution)

In this article I will describe my solution to the DS Bowl 2018 and why it was a lottery and post a link to my dockerized solution Статьи автора - Блог -

snakers4 (Alexander), April 15, 09:49


(0), Leslie N. Smith US Naval Research Laboratory

(1) Will serve as a good intuition starter if you have little experience (!)

(2) Some nice ideas:

- The test/validation loss is a good indicator of the network’s convergence - especially in early epochs

- The amount of regularization must be balanced for each dataset and architecture

- The practitioner’s goal is obtaining the highest performance while minimizing the needed computational time

(smaller batch - less stability and faster convergence)

- Optimal momentum value(s) will improve network training

(3) The author does not study the difference between SGD and Adam in depth =( Adam kind of solves much of his pains

(4) In my practice the following approach works best:

- Aggressive training with Adam to find the optimal LR

- Apply various LR decay regimes to determine the optimal

- Use low LR or CLR in the end to converge to a lower value (possible overfitting)

- Test on test / delayed test end-to-end

- In my experience - a strong model with good params will start with test/val set loss much lower / target metric much higher than on the train set

- In some applications if your CNN is memory intesive - you just opt for the largest batch possible (usually >6-8 works)

- Also there is no mention of augmentations - they usually help reduce overfitting much better than hyper parameters


Nice read about systemctl


How To Use Systemctl to Manage Systemd Services and Units | DigitalOcean

Systemd is an init system and system manager that is widely becoming the new standard for Linux machines. While there is considerable controversy as to whether systemd is an improvement over the init systems it is replacing, the majority of distributi

snakers4 (Alexander), April 15, 08:06

2018 DS/ML digest 8

As usual my short bi-weekly (or less) digest of everything that passed my BS detector

Market / blog posts

(0) about the importance of accessibility in ML -

(1) Some interesting news about market, mostly self-driving cars (the rest is crap) -

(2) US$600m investment into Chinese face recognition -

Libraries / frameworks / tools

(0) New 5 point face detector in Dlib for face alignment task -

(1) Finally a more proper comparsion of XGB / LightGBM / CatBoost - (also see my thoughts here

(3) CNNs on FPGAs by ZFTurbo



(4) Data version control - looks cool



-- but I will not use it - becasuse proper logging and treating data as immutable solves the issue

-- looks like over-engineering for the sake of overengineering (unless you create 100500 datasets per day)


(0) TF Playground to seed how simplest CNNs work -


(0) Looks like GAN + ResNet + Unet + content loss - can easily solve simpler tasks like deblurring

(1) You can apply dilated convolutions to NLP tasks -

(2) High level overview of face detection in -

(3) Alternatives to DWT and Mask-RCNN / RetinaNet?

- Has anybody tried anything here?


(0) A more disciplined approach to training CNNs - (LR regime, hyper param fitting etc)

(1) GANS for iamge compression -

(2) Paper reviews from ODS - mostly moonshots, but some are interesting



(3) SqueezeNext - the new SqueezeNet -




snakers4 (Alexander), April 12, 08:08

DS Bowl 2018 stage 2 data was released.

It has completely different distribution from stage 1 data.

How do you like them, apples?

Looks like Kaggle admins really have no idea about dataset curation, or all of this is mean to misguide manual annotators.

Anyway - looks like random bs.



snakers4 (Alexander), April 10, 09:01

A bit more on semantic segmentation, now 3D

{V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation


--> Link / authors, Fausto Milletari / Nassir Navab / Seyed-Ahmad Ahmadi

--> Essence:

(0) Essentially applies UNet to 3D with a custom DICE based loss

(1) Architecture - - basically UNet with 3D convolutions. Upsampling / downsampling -

(2) PReLu (no ablation test)

(3) Receptive fields of layers -

(4) 3D DICE loss - (wo BCE?)

--> The paper does not use all the juice possible - hacky transfer learning (obvious idea - just stacking Imagenet filters), CLR, LinkNet architectures, etc

--> Looks like a good baseline / reference

{An application of cascaded 3D fully convolutional networks for medical image segmentation


-->, a group of Japanese researchers

--> Essence:

(0) 2 stage 3D UNet, ablation test against 2D FCNs

(1) Loss - 3D cross-entropy

(2) Transfer learning - it works for other datasets, give a mild boost (1-3 %)

(3) 80-90% DICE, varies by organ

(4) weights downloadable (Caffe...)

--> Essentially a 2 stage process is dictated by memory considerations:

(0) Pipeline

In the long run transfer learning may rule, but here legal limitations may slow down this process.



snakers4 (Alexander), April 08, 13:48

As you may know (for newer people on the channel), sometimes we publish small articles on the website.

This time it covers a recent Power Laws challenge on DrivenData, which at first seemed legit and cool, but in the end turned back into a pumpkin.

Here is an article:





Playing with electricity - forecasting 5000 time series

In this article I share our experience participating in a recent time series challenge on Drivendata and my personal ideas about ML competitions Статьи автора - Блог -

snakers4 (Alexander), April 07, 11:14

The trend for smaller / inadequate prizes and weird datasets continues:

- This years' DS Bowl on Kaggle features a small public train dataset (650 images) vs. much larger delayed validation dataset (3000 images). Cheap annotation, anyone? =) Also remarkably, for a somewhat difficult task (instance segmentation) - the prize is much lower than the last year;

- New autonomous driving contest on Kaggle, as well as other CVPR competitions - features extremely large datasets, extremely low prizes (US$1-2k), and no travel costs to CVPR covered. Ofc you can win and fly there, but this will not even cover your GPU costs;

- The recent xnView challenge I really wanted to participate - requires a US Tax number to be eligible for prizes. Of course they do not know about double taxation treaties and WEP-8 tax exemptions;

Alas =(


snakers4 (Alexander), April 05, 06:19

For some reason when you use PyTorch multi-thread data loaders, it stalls if you use OpenCV and not set


Nice to know this.


snakers4 (Alexander), April 03, 07:53

Handy opencv snippet to transform grayscale mask with labels (i.e. 0 for background, 1,2,3 etc) into a colourful map

y_pred_coloured = cv2.applyColorMap((y_pred / y_pred.max() * 255).astype('uint8'), cv2.COLORMAP_JET)


snakers4 (Alexander), March 30, 04:35

New TF release (if you are into it, I personally use TensorBoard)


What is interesting

TensorBoard Debugger Plugin, the graphical user interface (GUI) of TensorFlow Debugger (tfdbg), is now in alpha.



tensorflow - Computation using data flow graphs for scalable machine learning

A new post by Fchollet


What worries me about AI

Disclaimer: These are my own personal views. I do not speak for my employer. If you quote this article, please have the honesty to present…

snakers4 (Alexander), March 20, 13:40

A video about realistic state of chat-bots (RU)




Neural Networks and Deep Learning lab at MIPT

При звонке в сервис или банк диалог с поддержкой строится по одному и тому же сценарию с небольшими вариациями. Отвечать «по бумажке» на вопросы может и не человек, а чат-бот. О том, как нейронные...

snakers4 (Alexander), March 14, 09:53

A nice thread about ML reproducibility


[D] Are the hyper-realistic results of... • r/MachineLearning

**Tacotron-2:** **Wavenet:**...

snakers4 (Alexander), March 10, 13:59

Interesting / noteworthy semseg papers

In practice - UNet and LinkNet are best and simple solutions.

Rarely people report that something like Tiramisu works properly.

Though I saw once in last Konika competition - a good solution based on DenseNet + Standard decoder.

So I decided to read some of the newer and older Semseg papers.

Classic papers

UNet,LinkNet - nuff said

(0) Links

- UNet -

- LinkNet -

Older, overlooked, but interesting papers

SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation

One of original papers before UNet


(1) Basically UNet w/o skip connections but it stores pooling indices

(1) SegNet uses the max pooling indices to upsample (without learning) the feature map(s) and convolves with a trainable decoder filter bank

ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation

Paszke, Adam / Chaurasia, Abhishek / Kim, Sangpil / Culurciello, Eugenio

(0) Link


(1) Key facts

- Up to 18× faster, 75× less FLOPs, 79× less parameters vs SegNet or FCN

- Supposedly runs on NVIDIA Jetson TX1 Embedded Systems

- Essentially a minzture of ResNet and Inception architectures

- Overview of the architecture



(2) Interesting ideas

- Visual information is highly spatially redundant, and thus can be compressed into a more efficient representation

- Highly assymetric - decoder is much smaller

- Dilated convolutions in the middle => significant accuracy boost

- Dropout > L2

- Pooling operation in parallel with a convolution of stride 2, and concatenate resulting feature maps

Newer papers

xView: Objects in Context in Overhead Imagery - new "Imagenet" for satellite images

(0) Link

- Will be available here

(1) Examples



(2) Stats

- 0.3m ground sample distance

- 60 classes in 7 different parent classes

- 1 million labeled objects covering over 1,400 km2 of the earth’s surface

- classes

(3) Baseline

- Their baseline using SSD has very poor performance ~20% mAP

Rethinking Atrous Convolution for Semantic Image Segmentation

(0) Link


- Liang-Chieh Chen George / Papandreou Florian / Schroff Hartwig / Adam

- Google Inc.

(1) Problems to be solved

- Reduced feature resolution

- Objects at multiple scales

(2) Key approaches

- Image pyramid (reportedly works poorly and requires a lot of memory)

- Encoder-decoder

- Spatial pyramid pooling (reportedly works poorly and requires a lot of memory)

(3) Key ideas

- Atrous (dilated) convolution -

- ResNet + Atrous convolutions -

- Atrous Spatial Pyramid Pooling block -

(4) Performance

- As with the latest semseg methods, true performance boost is unclear

- I would argue that such methods may be useful for large objects



snakers4 (Alexander), March 09, 05:49

PyTorch caught up with keras


François Chollet

TensorFlow is the platform of choice for deep learning in the research community. These are deep learning framework mentions on arXiv over the past 3 months

snakers4 (Alexander), March 07, 06:03

New articles about picking GPUs for DL



Also most likely for 4-GPU set-ups in a case some of them may be required to have water cooling to avoid thermal throttling.


Picking a GPU for Deep Learning

Buyer’s guide at the beginning of 2018

snakers4 (Alexander), March 07, 05:03

2018 DS/ML digest 6


(1) A new amazing post by Google on distil -

This is really amazing work, but their notebooks tells me that it is a far cry from being able to be utilized by the community -

This is how the CNN sees the image -

Expect this to be packaged as part of Tensorboard in a year or so)


(1) New landmark dataset by Google - - looks cool, but ...

Prizes in the accompanying Kaggle competitions are laughable

Given that datasets are really huge...~300G

Also also if you win, you will have to buy a ticket to the USA on your money ...

(2) Useful script to download the images

(3) Imagenet for satellite imagery - - pre-register paper

(4) CVPR 2018 for satellite imagery -

Papers / new techniques

(1) Improving RNN performance via auxiliary loss -

(2) Satellite imaging for emergencies -

(3) Baidu - neural voice cloning -


(1) Google TPU benchmarks -

As usual such charts do not show consumer hardware.

My guess is that a single 1080Ti may deliver comparable performance (i.e. 30-40% of it) for ~US$700-1000k, i.e. ~150 hours of rent (this is ~ 1 week!)

Miners say that 1080Ti can work 1-2 years non-stop

(2) MIT and SenseTime announce effort to advance artificial intelligence research

(3) Google released its ML course - - but generally it is a big TF ad ... Andrew Ng is better for grasping concepts


(1) Interesting thing - all ISPs have some preferential agreements between each other -




The Building Blocks of Interpretability

Interpretability techniques are normally studied in isolation. We explore the powerful interfaces that arise when you combine them -- and the rich structure of this combinatorial space.

snakers4 (Alexander), March 06, 14:34

I have seen questions on forums - how to add Keras-like progress bar to PyTorch for simple models?

The answer is to use tqdm and this property


This example is also great

from tqdm import trange

from random import random, randint

from time import sleep

t = trange(100)

for i in t:

# Description will be displayed on the left

t.set_description('GEN %i' % i)

# Postfix will be displayed on the right, and will format automatically

# based on argument's datatype

t.set_postfix(loss=random(), gen=randint(1,999), str='h', lst=[1, 2])




Can I add message to the tqdm progressbar?

When using the tqdm progress bar: can I add a message to the same line as the progress bar in a loop? I tried using the "tqdm.write" option, but it adds a new line on every write. I would like each

snakers4 (Alexander), February 23, 06:30

Was looking for CLAHE abstraction for my image pre-processing pipeline and found one in the Internet

class CLAHE:

def __init__(self, clipLimit=2.0, tileGridSize=(8, 8)):

self.clipLimit = clipLimit

self.tileGridSize = tileGridSize

def __call__(self, im):

img_yuv = cv2.cvtColor(im, cv2.COLOR_BGR2YUV)

clahe = cv2.createCLAHE(clipLimit=self.clipLimit, tileGridSize=self.tileGridSize)

img_yuv[:, :, 0] = clahe.apply(img_yuv[:, :, 0])

img_output = cv2.cvtColor(img_yuv, cv2.COLOR_YUV2BGR)

return img_output


snakers4 (Alexander), February 20, 04:16

So a couple of things -

- Movidius USB stick is enough to launch real-time object detection which is interesting to know

- It has shitty driver and library support (Caffe was mentioned)

- Installing everything is FAR from trivial (no idea why virtual box was used, but whatever)

- This guide uses Virtual box instead of Docker which says much

Also PyImageSearch is a sellout - he most likely has advertiser-friendly featured content in this post, looks like the Movidius stick topcoder event did not gain enough traction...

So - use Nvidia Jetsons for embedded solutions and do not bother with this. But it's good that new products emerge.


Real-time object detection on the Raspberry Pi with the Movidius NCS - PyImageSearch

In this tutorial I'll demonstrate how you an achieve real-time object detection on the Raspberry Pi using deep learning and Intel's Movidius NCS.

snakers4 (Alexander), February 19, 10:25

One more article about usual suspects when your CNN fails to train



37 Reasons why your Neural Network is not working

The network had been training for the last 12 hours. It all looked good: the gradients were flowing and the loss was decreasing. But then…

snakers4 (Alexander), February 18, 08:05

Even though I am preparing a large release on GAN application on real example, I just could not help sharing these 2 links.

They are just an absolute of perfection for GANs on PyTroch



Also this is the most idiomatic PyTorch code (Imagenet finetuning) code I have ever seen


So if you are new to PyTorch, then these links will be very useful)





Contribute to WassersteinGAN development by creating an account on GitHub.

Which of the latest projects did you like the most?

Still waiting for GANs – 17

👍👍👍👍👍👍👍 49%

Satellites! – 9

👍👍👍👍 26%

Nothing / not interested / missed them – 5

👍👍 14%

Jungle! – 3

👍 9%

PM me for other options – 1

▫️ 3%

👥 35 people voted so far.

snakers4 (Alexander), February 14, 11:48

2017 DS/ML digest 4

Applied cool stuff

- How Dropbox build their OCR - via CTC loss -

Fun stuff

- CNN forward pass done in Google Sheets -

- New Boston Robotics robot - opens doors now -

- Cool but toothless list of jupyter notebooks with illustrations and models

- Best CNN filter visualization tool ever -

New directions / moonshots / papers

- IMPALA from Google - DMLab-30, a set of new tasks that span a large variety of challenges in a visually unified environment with a common action space



- Trade crypto via RL -

- SparseNets? -

- Use Apple watch data to predict diseases

- Google - Evolution in auto ML kicks in faster than RL -

- R-CNN for human pose estimation + dataset

-- Website + video

-- Paper

Google's Colaboratory gives free GPUs?

- Old GPUs

- 12 hours limit, but very cool in theory



Sick sad world

- China has police Google Glass with face recognition

- Why slack sucks -

-- Email + google docs is better for real communication


- Globally there are 22k ML developers

- One more AI chip moonshot -

- Google made their TPUs public in beta - US$6 per hour

- CNN performance comparable to human level in dermatology (R-CNN) -

- Deep learning is greedy, brittle, opaque, and shallow

- One more medical ML investment - US$25m for cancer -




snakers4 (Alexander), February 14, 04:54

Article on SpaceNet Challenge Three in Russian on habrhabr - please support us with your comments / upvotes


Also if you missed:

- The original article

- The original code release

... and Jeremy Howard from retweeted our solution, lol



But to give some idea which pain the TopCoder platform induces on the contestants, you can read

- Data Download guide

- Final testing guide

- Code release for their verification process




Из спутниковых снимков в графы (cоревнование SpaceNet Road Detector) — попадание топ-10 и код (перевод)

Привет, Хабр! Представляю вам перевод статьи. Это Вегас с предоставленной разметкой, тестовым датасетом и вероятно белые квадраты — это отложенная валидация...

snakers4 (Alexander), February 13, 07:55

Interesting hack from n01z3 from ODS

For getting that extra 1%

Snapshot Ensembles / Multi-checkpoint TTA:


- Train CNN with LR decay until convergence, use SGD or Adam

- Use cyclic LR starting to train the network from the best checkpoint, train for several epochs

- Collect checkpoints with the best loss and use them for ensembles / TTA


Google TPUs are released in beta..US$200 per day?

No thank you! Also looks like only TF is supported so far.

Combined with rumours, sounds impractical.



Cloud TPU machine learning accelerators now available in beta

By John Barrus, Product Manager for Cloud TPUs, Google Cloud and Zak Stone, Product Manager for TensorFlow and Cloud TPUs, Google Brain Team...

snakers4 (Alexander), February 10, 07:13

So we started publishing articles / code / solutions to the recent SpaceNet3 challenge. A Russian article on will also be published soon.

- The original article

- The original code release

... and Jeremy Howard from retweeted our solution, lol



But to give some idea which pain the TopCoder platform induces on the contestants, you can read

- Data Download guide

- Final testing guide

- Code release for their verification process




How we participated in SpaceNet three Road Detector challenge

This article tells about our SpaceNet Challenge participation, semantic segmentation in general and transforming masks into graphs Статьи автора - Блог -

snakers4 (Alexander), February 08, 09:13 lesson 11 notes:

- Links

-- Video


- Semantic embeddings + imagenet can be powerful, but not deployable per se

- Training nets on smaller images usually works

- Comparing activation functions

- lr annealing

- linear learnable colour swap trick

- adding Batchnorm

- replacing max-pooling with avg_pooling

- lr vs batch-size

- dealing with noisy labels

- FC / max-pooling layer models are better for transfer-learning?

- size vs. flops vs. speed

- cyclical learning rate paper

- Some nice intuitions about mean shift clustering





Lesson 11: Cutting Edge Deep Learning for Coders
We’ve covered a lot of different architectures, training algorithms, and all kinds of other CNN tricks during this course—so you might be wondering: what sho...

Meta research on the CNNs

(also this amazing post

An Analysis of Deep Neural Network Models for Practical Applications

Key findings:

(1) power consumption is independent of batch size and architecture;

(2) accuracy and inference time are in a hyperbolic relationship;

(3) energy constraint = upper bound on the maximum achievable accuracy and model complexity;

(4) the number of operations is a reliable estimate of the inference time


- Accuracy and param number -

- Param efficiency -

Also a summary of architectural patterns



Deep Learning Scaling is Predictable, Empirically




- various empirical learning curves show robust power-law region

- new architectures slightly shift learning curves downwards

- model architecture exploration should be feasible with small training data sets

- it can be difficult to ensure that training data is large enough to see the power-law learning curve region

- irreducible error region

- each new hardware generation with improved FLOP rate can pro- vide a predictable step function improvement in relative DL model accuracy



Neural Network Architectures

Deep neural networks and Deep Learning are powerful and popular algorithms. And a lot of their success lays in the careful design of the…

snakers4 (Alexander), February 07, 14:09

Following our blog post, we also posted a Russian translation of the Jungle competition to habrhabr




Соревнование Pri-matrix Factorization на DrivenData с 1ТБ данных — как мы заняли 3 место (перевод)

Привет, Хабр! Представляю вашему вниманию перевод статьи "Animal detection in the jungle — 1TB+ of data, 90%+ accuracy and 3rd place in the competition". Или...

snakers4 (Alexander), February 06, 05:23

We are starting to publish our code / solutions / articles from recent competitions (Jungle and SpaceNet three).

This time the code will be more polished / idiomatic, so that you can learn something from it!

Jungle competition

- Finally it was verified that we indeed won the 3rd place)


Blog posts



- An adaptation for will be coming soon

Code release and architecture:

- Code

- Architecture

-- 1st place (kudos to Dmytro) - simple and nice

-- Ours

-- 2nd place - 4-5 levels of stacking

Please comment under posts / share / buy us a coffee!

- Buy a coffee

- Rate our channel tg://resolve?domain=tchannelsbot&start=snakers4




Pri-matrix Factorization

Chimp&See has collected nearly 8,000 hours of footage reflecting chimpanzee habitats from camera traps across Africa. Your challenge is to build a model that identifies the wildlife in these videos.

snakers4 (Alexander), February 05, 14:57

We also managed to get into top-10 in SpaceNet3 Road Detection challenge


(Final confirmation awaits)

Here is a sneak peak of our solution


A blog post + repo will follow





Flowchart Maker & Online Diagram Software is a free online diagramming application and flowchart maker . You can use it to create UML, entity relationship, org charts, BPMN and BPM, database schema and networks. Also possible are telecommunication network, workflow, flowcharts, maps overlays and GIS, electronic circuit and social network diagrams.