[Scikit-learn-general] GSoC 2012

[Bala Subrahmanyam Varanasi]

Dear all,

I would like to participate in Google Summer of Code this year. Please let
me know the ideas which you would like to implement in scikit-learn in GSoC
- 2012.

Also... I'm attending to Stanford's Online courses - ML class and NLP
class. I believe this is the right time to discuss. Because, I can learn
new things before the start of GSoC and can work on challenging
implementations in scikit-learn.

Thank you.

Bala Subrahmanyam Varanasi
IV B.Tech, Information Technology
Vishnu Institute of Technology
e-mail: ***@gmail.com
contact number: [phone]

[Andreas]

Hi Bela.
I'm not sure how this usually goes but here is my current wish list.
We'd have to discuss whether any of that actually fits into the scikits,
thou ;)

- Multilayer Perceptron and Multinomial Logistic regression
I have been working on that so maybe there is not enough
left to do there for a GSoC. Not sure, though

- Graph Cut Energy minimization
This is an inference technique so I'm not totally sure
if this should go into scikit-learn. Could also be a
candidate for scikit-image.
The main work would be to implement an efficient
max flow algorithm and then do graph constructions
for alpha expansion and alpha-beta swaps.

- Averaged gradient descent
I think this is on everybody's wish list. Not
sure how much work this will be.
I'm sure lot's of people will have to say something to that ;)
See issue #543: https://github.com/scikit-learn/scikit-learn/issues/543

- Structured SVM / CRF learning
This is a big one. Not sure what other people think of it.
I think having a structured SVM would be great.
At the moment, the most commonly used implementation
is Joachim's SVMstruct.
This has licensing issues but talking to him might help.
Another option is implementing optimization via SGD
or, if you want to go crazy, cutting plane techniques
or bundle methods yourself.
Designing the interface is also non-trivial.
One would have to think about whether / how it
is possible to use structured SVMs just from Python,
without writing Cython functions.

- Low rank kernel approximations (Nystrom methods)
This is mainly interesting for SVMs.
The idea is to approximate the kernel matrix with
a low rank factorization and use this to construct
a linear SVM problem.
This is related to the current kernel approximation
module but has a somewhat other approach.
This method makes large scale SVMs fast / possible

- Kernel Perceptron
There is a (I think) pure Python implementation
by Mathieu that could be Cythonized.


That's it for the moment, I think.
I'd be happy to mentor any of the above projects
if the others agree that they are sensible.

Maybe we should update the wiki for the next GSoC?

Cheers,
Andy

Post by Bala Subrahmanyam Varanasi
Dear all,
I would like to participate in Google Summer of Code this year. Please
let me know the ideas which you would like to implement in
scikit-learn in GSoC - 2012.
Also... I'm attending to Stanford's Online courses - ML class and NLP
class. I believe this is the right time to discuss. Because, I can
learn new things before the start of GSoC and can work on challenging
implementations in scikit-learn.
Thank you.
Bala Subrahmanyam Varanasi
IV B.Tech, Information Technology
Vishnu Institute of Technology
contact number: [phone]

[Lars Buitinck]

Post by Andreas
- Structured SVM / CRF learning
    This is a big one. Not sure what other people think of it.
    I think having a structured SVM would be great.

+100 on this one...

Post by Andreas
    Designing the interface is also non-trivial.

Indeed. I suspect different APIs would be needed for different cases
(linear-chain case, tree case, general case). Having just one of these
would be great, but I agree this might be a *big* project.

One more thing one my wishlist is semisupervised meta-algorithms, like
self-training, co-training, co-boosting. These should not be
incredibly hard to implement, but they're still far from trivial.

[Gael Varoquaux]

Post by Lars Buitinck

Post by Andreas
- Structured SVM / CRF learning
    This is a big one. Not sure what other people think of it.
    I think having a structured SVM would be great.

+100 on this one...

For this, do we need to have our own SVM solver? This is a naive
question, I have never looked at structured SVM.

This seems to me as a fairly challenging project.

Post by Lars Buitinck
One more thing one my wishlist is semisupervised meta-algorithms, like
self-training, co-training, co-boosting. These should not be
incredibly hard to implement, but they're still far from trivial.

Yes, I think that having a good semi-supervised codebase (including model
selection) would be a interesting project. What I like about it, is that
it seems to me as having a gradual difficulty.

Of course, other suggestions are welcome. Maybe we should start a wiki
page.

Gael

[Andreas]

Post by Gael Varoquaux

Post by Lars Buitinck

Post by Andreas
- Structured SVM / CRF learning
This is a big one. Not sure what other people think of it.
I think having a structured SVM would be great.

+100 on this one...

For this, do we need to have our own SVM solver? This is a naive
question, I have never looked at structured SVM.
This seems to me as a fairly challenging project.

This is quite definitely a challenging project.
This should only be given to someone with a fair understanding
of the topic.

There are several options, as I tried to say in my initial post:
1) bindings for an existing structured SVM.
2) bindings for a smart solver with structured svm code by us
3) using SGD for solving. This means "having our own SVM solver"
-- but we already got one in SGDClassifier.
4) write a solver using cutting plane or bundle methods (not quite
sure if this is a good idea)

[Gael Varoquaux]

Having this feature might get us a LOT of attention.
But this is really not a simple project.

Before trying to jump to the super fancy features, I'd rather have a
polished and versatile version of the scikit. They are many things that I
find that we haven't explored right. For instance these are my personal
pain points:

* we don't have an online learning framework.

* Our model selection framework is still weak

- see
https://github.com/scikit-learn/scikit-learn/pull/443#issuecomment-3231270

- also, it the difficulty to do nested cross-validation with a specific
cross-validation strategy,

* we are light on the semi-supervised API

* our parameter naming is not uniform-enough across models.

All these are points that I'd like to see addressed, because I fear that
they could all induce a change in API or conventions. And I'd like API
and conventions to be stabilized, to be able to push out a 1.0 (I am
talking 6 months to 1 year horizon).

G

PS: On the down side, I am having crazy days these last weeks. I spend
the whole day at work looking at other people's problem, and when I get
home in the evening, instead of hacking on open source, I answer email
and try to get done the work that should have done during the day...
Sorry lads, just mumbling at my own improductivity.

[Andreas]

Post by Gael Varoquaux

Having this feature might get us a LOT of attention.
But this is really not a simple project.

Before trying to jump to the super fancy features, I'd rather have a
polished and versatile version of the scikit.

I totally agree - I tried to do as much polishing as I can the
last couple of weeks.
There is still a lot to do. I opened some issues today and
yesterday to track stuff that seemed important to me.

I have no experience with GSoC and I will totally bow
to you wisdom there. My thinking was that single
algorithms are more "project-like" than doing polishing here and
there.

There is important refactoring being done by Lars and Mathieu
at the moment which is really great. But I wouldn't give that
to someone as a project.

Post by Gael Varoquaux
They are many things that I
find that we haven't explored right. For instance these are my personal
* we don't have an online learning framework.
* Our model selection framework is still weak
- see
https://github.com/scikit-learn/scikit-learn/pull/443#issuecomment-3231270
- also, it the difficulty to do nested cross-validation with a specific
cross-validation strategy,
* we are light on the semi-supervised API
* our parameter naming is not uniform-enough across models.
All these are points that I'd like to see addressed, because I fear that
they could all induce a change in API or conventions.

I noticed some cross-validation issues but not all that you mentioned.
We should maybe plan a bit more on that.

About online and semi-supervised learning:
I feel these are two specific sub-fields that many people are interested
in but that are not central to machine learning.
I am not sure I would want the scikit api to focus on these.
If you go to a machine learning conference, I'm pretty sure
there will be more people working on structured learning
than on semi-supervised and online learning.

Don't get me wrong. I don't want to quickly forcestructured
learning into the scikit. It is a long term goal of me to
have this in a nice accessible form. I just wanted to mention
it as an option.

Post by Gael Varoquaux
And I'd like API
and conventions to be stabilized, to be able to push out a 1.0 (I am
talking 6 months to 1 year horizon).

I couldn't agree more!

Cheers,
Andy

[Gael Varoquaux]

Post by Andreas
I have no experience with GSoC and I will totally bow
to you wisdom there. My thinking was that single
algorithms are more "project-like" than doing polishing here and
there.

Yes. My point was that I'd like to see project that help us close these
gaps, rather than open new ones.

Multi-task learning or semi-supervised learning are two projects that
would probably help us find the limits of our cross-validation scheme.
Structured output, as much as I personnally am interested in it, seems to
be of the kind to open a gap, and it should probably be tackled by
someone with good experience with the scikit.

Vlad's idea of exploring trace norm would be interesting, as it would be
relevent for both multi-task and unsupervised learning.

G

[Mathieu Blondel]

On Thu, Jan 19, 2012 at 7:44 AM, Gael Varoquaux

Post by Gael Varoquaux

Having this feature might get us a LOT of attention.
But this is really not a simple project.

Before trying to jump to the super fancy features, I'd rather have a
polished and versatile version of the scikit. They are many things that I
find that we haven't explored right. For instance these are my personal
 * we don't have an online learning framework.
 * Our model selection framework is still weak
   - see
     https://github.com/scikit-learn/scikit-learn/pull/443#issuecomment-3231270
   - also, it the difficulty to do nested cross-validation with a specific
      cross-validation strategy,
 * we are light on the semi-supervised API
 * our parameter naming is not uniform-enough across models.

Also the scikit has a bias towards dense data. It would be nice if
more estimators could work with sparse data too.

Mathieu

[Peter Prettenhofer]

Post by Andreas

Post by Gael Varoquaux

Post by Lars Buitinck
[..]

Post by Andreas
- Structured SVM / CRF learning
     This is a big one. Not sure what other people think of it.
     I think having a structured SVM would be great.

+100 on this one...

For this, do we need to have our own SVM solver? This is a naive
question, I have never looked at structured SVM.
This seems to me as a fairly challenging project.

This is quite definitely a challenging project.
This should only be given to someone with a fair understanding
of the topic.
1) bindings for an existing structured SVM.
2) bindings for a smart solver with structured svm code by us
3) using SGD for solving. This means "having our own SVM solver"
-- but we already got one in SGDClassifier.
4) write a solver using cutting plane or bundle methods (not quite
sure if this is a good idea)
 

[Alexandre Gramfort]

i've created the wiki page to organize what was suggested and so
people can volunteer for mentoring.

https://github.com/scikit-learn/scikit-learn/wiki/A-list-of-topics-for-a-google-summer-of-code-%28gsoc%29-2012

Alex

On Thu, Jan 19, 2012 at 8:38 AM, Peter Prettenhofer

Post by Andreas

Post by Gael Varoquaux

Post by Lars Buitinck
[..]

Post by Andreas
- Structured SVM / CRF learning
     This is a big one. Not sure what other people think of it.
     I think having a structured SVM would be great.

+100 on this one...

For this, do we need to have our own SVM solver? This is a naive
question, I have never looked at structured SVM.
This seems to me as a fairly challenging project.

This is quite definitely a challenging project.
This should only be given to someone with a fair understanding
of the topic.
1) bindings for an existing structured SVM.
2) bindings for a smart solver with structured svm code by us
3) using SGD for solving. This means "having our own SVM solver"
-- but we already got one in SGDClassifier.
4) write a solver using cutting plane or bundle methods (not quite
sure if this is a good idea)