Proposal Excellenzzzzzzzzz

When I am asked to review a grant proposal by a program director at NSF (doesn't matter which program), I almost always say yes. There have been a few cases that I can remember when I was sent a proposal quite far out of my field of expertise, and I said no to those, but it is rare that I am asked to review proposals that I have no reasonable basis to judge. This is not because I am so awesomely expert in so many things but because the program directors do an excellent job of targeting reviewers.

Of course, saying yes to review a proposal on a topic that you find somewhat-to-very interesting doesn't mean the proposal will be good (or very good). And, unless you have reviewed proposals and/or manuscripts by the PI/coPIs before, you may not know in advance whether it is likely to be well written or not.

So it can be hard to predict what the review-experience is going to be like, even if you have some basic information at the time of agreeing to the review. I am thinking in particular of a proposal that I reviewed not long ago. The proposal was on an interesting topic, the writing was mostly good in a technical sense, and the proposed work involved what seemed to be the appropriate approach (methods etc.), and yet.. it was extremely difficult for me to read more than a page without flinging it aside (or, in reality, the electronic equivalent of flinging, as I was reading it as a pdf).
Why? Because it was so so so so boring to read this proposal. It was very painful.

Is my attention span getting shorter as I get older and busier and more jaded? Do I need lots of insightful illustrations and formatted text telling me what is important? Do I have to be grabbed in the project summary and/or first paragraph of the project description or else I have trouble being interested in the proposal? Answers at the end of this post.

Eventually I did read the proposal all the way through, of course, and I even re-read what I thought were some critical parts, but it was a slog from start to finish. I had to work really hard to figure out that the proposed work was of some interest and significance. The PI did not highlight the significance, but it was in there, somewhere.
I have written about related issues before: How hard do/should we as reviewers work to see the good (or excellence) in a proposal that has some technical flaws? In many cases, the flaws are in the writing, but in this case it wasn't so much how the sentences were constructed (each one by itself was fine), but in how the proposal as a whole was framed. If I had to work so hard to extract the interesting essence of the proposal, does that mean I will/should rate the proposal lower than I would if the proposal were technically excellent (in addition to containing interesting proposed work)?

(I am ignoring, in this particular discussion, the issue of giving early-career people a bit of a break; i.e., not expecting proposal perfection for those who haven't written many or any before.)

I fear the answer is yes, in a way. There was no way I was giving this proposal a rating of Excellent (the highest rating for an NSF proposal). If the proposal itself had been really well put together, showing more vision in general about the work and how to explain it, it could possibly have been in the Excellent range, but given what I saw in the proposal: no. Ignoring the fact that some of the shortcomings in understanding the proposal may have been my own, can I assume that the dense, complicated, non-linear, dry presentation of the proposed work is an indication of how the research itself will be undertaken, understood, and communicated? I decided the answer to that was also yes, and hence my overall positive but not-Excellent review.

That might sound harsh, so let me say that my first impression of the proposal was even more negative, but my opinion improved with more careful reading and thought. The proposed work is good, even very good. In the end, my rating was quite positive, and (I hope) backed up by what I wrote in my review. So:

Is my attention span getting shorter as I get older and busier and more jaded? Yes

Do I need lots of insightful illustrations and formatted text telling me what is important? Maybe -- it would be nice, anyway. Of course, writing "The hypothesis to be tested is ..." in bold italics underlined and set off with an indented bullet, surrounded by spaces and maybe some nice subtle shading and prominent on page 1 of a proposal doesn't guarantee you won't get a reviewer who writes "It is not clear to me what hypothesis is being tested", but one can reduce the chances of getting comments like this.

Do I have to be grabbed in the project summary and/or first paragraph of the project description or else I have trouble being interested in the proposal? Not this time, but I think in general, yes. I eventually forced myself to sit down and read the whole proposal carefully, painful though it was to do so, and saw some interesting things buried in the dense text, but I wouldn't want to risk this myself when writing a proposal. Perhaps my review would have been more positive if I hadn't found it a huge chore to read the proposal. An annoyed, weary, bored reviewer is probably best avoided.

Useless Moneybags

Every once in a while, someone makes a statement in a comment about how their advisor "doesn't do anything". In some cases, there is the added description that an advisor "doesn't do anything except get grants". I find these comments fascinating, but not necessarily in a good way. I am sure I have written about this before, but since I was thinking about the phenomenon recently as I was sitting suspended over an ocean, I am writing about it again now.

I am sure there are exceptions -- of course I don't know how things work in other fields or at other institutions other than ones with which I am closely associated -- but I think in many cases, this comment displays a misconception about the definition of "anything" in a research context. [This is where the young and perhaps not-so-young say to their advisors (but perhaps not aloud): "But that's your fault because you are supposed to teach us what you do". Yes, that's true to some extent, although it's a rather lame reason for remaining ignorant over the entire course of a graduate program, as I've discussed before.]

In any case, I know that the Do-Nothing (except provide grants) Advisors are believed to exist, and that is why many of us learn during fascinating and mandatory Research Ethics workshops that just providing the money for a research project is not sufficient justification for us to be a co-author on the resulting papers. And yet, I always wonder: But what if the research was our IDEA? Doesn't that count?

Today, what I want to know is how many readers have said or thought that their advisor doesn't (or didn't) do anything, meaning in this case that the advisor doesn't (or didn't) do any research (whether it was true or not)? If anyone leaves a comment confessing to having this thought/belief, it would also be helpful to know the academic discipline involved. In my field, it's relatively easy for me to do some actual research myself, but in other fields or in other research group configurations, it may be more difficult for an advisor to do this. Hence, additional information may be important for exploring and understanding this phenomenon.

There may be various modes of thought that feed into such a view. One that I imagine is common goes something like this:

- because you and other students ± postdocs, techs etc. are the ones actually generating data, you are the ones doing the real work, and your advisor is therefore "not doing anything".

But I hope it is more complex than that, and not an indication of a lack of appreciation for the value of ideas -- the ideas that can lead to a successful proposal and therefore a grant, the ideas for overcoming obstacles that may arise during the data-gathering stages, and the ideas that come once the data (or whatever) are obtained and it's time to think about the results, understand them, discuss them, interpret them, and thereby generate new ideas.

If a grad student who thinks their advisor doesn't do anything is in a situation in which they (the student) had some ideas that formed the core of a grant proposal that they largely wrote (perhaps with some help with the logistics of writing/submitting a proposal), got the grant (perhaps with their advisor's name on it), carried out the research largely independently (perhaps after learning some key techniques from someone other than the advisor), made the most significant interpretations, and wrote the papers, then go ahead and say it: your advisor didn't do much, if anything.

Otherwise, I think it is a strange and incorrect thing to say.

x% of Infinite

Those of us who got some "stimulus" funding (ARRA) in the form of a research grant a few years ago have to do quarterly reporting on the progress of this research, as opposed to annual reporting for a regular grant; in my case, the grant was from NSF. This quarterly reporting is not arduous; it just takes a few minutes of checking the info generated automatically (e.g., number of persons employed by the grant) and updating some text about research activities.

I have no idea who, if anyone, reads these quarterly reports. Unlike annual reports, which have to be approved by a program officer (at least at NSF, I don't know about other funding agencies), once these quarterly reports are done, they seem to be done.

The one part of the reporting that always makes me have to think a bit is the part where I have to report the % of the research that is completed. In some senses, of course, a research project is never really done. You can always do more, and then some more. Particular research questions lead to more research questions which lead to .. etc. That's one thing that is so great about research.

Realistically, though, a grant has a finite time span, so things do come to an end (financially). Therefore, I could answer the question in terms of "How much money is left in the grant?". That is a number that, in theory, can be determined well enough for reporting requirements, although never exactly at any given moment, given the complexity and vagaries of my university's accounting system.

Another way to phrase the question is:"What % of what you said you would do in the grant proposal is done at this time?" That's a tricky question for research projects that veer -- for scientifically valid reasons -- from what was proposed in the proposal. I have written about this before: my research group's grant proposals are our best guess for what we will do to solve the questions and problems posed, but, once the research starts and we get some results, we may find a different/better way to approach these problems, at least in some ways.

In that sense, the % completed of what we proposed to do may be a very different number from the % of the project that will be "completed" by the time the funding runs out.

I have been thinking about this over the last few years as I do my quarterly reports and have to select an answer from a pull-down menu with possible responses to the % completed question. For a long time, I was answering with low numbers because we were in the data-gathering stage (for well over half the duration of the grant), but once we were >>50% in terms of time and money left on the grant, I realized that the % completed should probably kick up a bit, even if we still have a lot more to do.

I suppose at some point I will have to say that the project is 100% completed; that is, when the money runs out, even if the research is not really completed. For accounting purposes, this is probably fine, but it will not be entirely true.

It Hurts When I Do This

Today in Scientopia, I respond to a reader's distress about re-reading a recently submitted grant proposal.

n proposals

In yesterday's post, there were some comments about how many proposals an individual faculty writes or "should" write in a year. Of course, there is no one answer to how many proposals one "should" write, even within a particular field.

We should each write and submit as many proposals as we can reasonably manage with our other responsibilities, taking into account also

• funding rate, which is no doubt quite low,
• amount of typical grants,
• needs of the research group (e.g., number of advisees and others to support),
• types and sizes of collaborations involved in research,
  
• career stage (tenured?),
• time-management skills and proposal-writing ability/style of the PI,
• number of other important responsibilities in the weeks before a proposal deadline,
• health/family situations, and
• expectations of the job/institution (including whether you have to raise some of your salary), and
• number of awesome ideas for transformative research.
  

Did I leave out any major ones?

There is no point in stating, as one commenter did, that if you write n proposals (say, n = 3), you are shirking your other responsibilities, such as advising students and writing papers. Many of us routinely submit > 3 proposals in a year and manage to get other things done as well. We are neither superhuman or super-irresponsible; this is just the way things work for some people, in some fields, on this very planet.

If, however, you totally shut down a month before a proposal deadline and do nothing else but work on that proposal, and you do that 3-4 times/year, then OK, you can say that you cannot write 3 or more proposals in a year and still get other important things done. But other people can.

If any one proposal has a not-great chance of being funded, and if not being funded is not a good option, then you have to find ways to write lots of (excellent) proposals and do everything else.

In the past academic year, I submitted 4 proposals, 3 as PI; 3 were funded, 2 with me as PI. I got other things done as well. I didn't expect to get so many of these proposals funded, and since I have some existing grants as well, I am taking a break from proposal-writing. It won't be a long break, however, because soon it will be time to try to get support for new projects and people.

As I rummaged in the FSP archives, vaguely recalling an earlier post on a similar topic, I was interested to read that in 2008, I stated that

I have been PI on 3 ± 1 grants (+ others as co-PI) at a time..for most of the 21st century..

This is still reasonably true. I may have reached a mid-career steady state.

But enough about me. If your job involves proposal-writing, how many proposals do you write in a typical year?

In coming up with a number, I suggest combining proposals submitted as PI and co-PI but not count those involving only having a minor role as subcontractor or senior personnel unless you had a major role in all phases of proposal-writing. I realize that some co-PIs (or even PIs) don't play much of a role in proposal-writing as well, so just use your discretion in coming up with the number. You can count both external and internal proposals, but count the latter only if they require significant effort. I am trying to get a sense for how many substantial proposals FSP readers typically write (not just put their name on) in a year.

How many proposals do you write in a typical year?
	0
	1
	2-3
	4-5
	6-7
	more than 7
pollcode.com free polls

In the comments, it would be interesting if you provided additional data; e.g. specify n proposals, your field, and job title or description. What is your personal funding success rate?

And, if anyone still has time despite needing to get back to writing proposals, it would also be interesting to know if you feel you are writing as many proposals as you can or should. Would you write more proposals if you had the time?

Due or Overdue?

The blogosphere has done a rather thorough job of thrashing the recent *scathing* report by Senator Coburn (R-Oklahoma) on supposed mismanagement and waste by the NSF. I will therefore quell the urge to point out how poorly researched, written, and organized the report it. It's a good thing the report wasn't funded by NSF or it would indeed be an excellent example of wasted effort and money by that agency.

I was, however, bothered by a particular section that has not received as much attention: the part about the supposedly shocking number of late and never-turned-in project reports:

A 2005 audit found that “[a]pproximately 47 percent of the 151,000 final and annual project reports required in the past 5 years were submitted late or not at all.”

There is a handy chart, organized by Directorate, showing the % final reports submitted on time, late, or not at all. I have some points to make about this:

1 - It is not surprising that so many reports are late. Reports are due 90 days before the end of the budget period, and at that time some of us are still scrambling to submit additional grant-related publications to include in the report. I would rather submit a late but more impressive report than an on-time report with less substance. One of the main motivators for me to file my reports (annual or final) sooner rather than later is that it is necessary to be up-to-date with reports in order for the paperwork for new grants to be processed, but if there is no pressure from that, I will wait until I can write a more complete and informative report.

2 - According to the chart in Coburn's document, although there are many late reports, the number of missing (never submitted) reports is low. It is misleading to combine late + missing reports and then focus on that number as if there is a major problem.

3 - The reports are administratively important for NSF and are one way that the programs/directorates can see what has been produced from grants (publications, education, outreach etc.), but the reports are not the actual "products" of the research and therefore are an imperfect measure of project success. It would be more relevant (but perhaps more difficult) to look at the actual outcomes of a grant -- as opposed to whether a project report has been filed on time -- and to look at contributions and impact beyond the expiration date of the grant.

I also wondered about the numbers related to unspent grant money apparently lying around, uncollected by NSF. Does this include money that PIs retain as part of no-cost extensions that give more time to do the research, or is that apparently leftover money accounted for because the expiration date of the grant changes when a no-cost extension is approved? If you only consider the original expiration date of the grant, it would look like there is unspent grant money. I couldn't tell from the report whether the unspent money was real or not.

And did anyone else think it was strange, given Coburn's concern for duplication and overlap among programs that fund STEM education for underrepresented populations, that the subtitle of the report, "Under the Microscope", is also the name of an NSF-funded website created by The Feminist Press to encourage girls and women to be involved in STEM fields? How do we all feel about this duplication of titles? I don't have any real data, but I am sure I am against it.

Responsible Conduct of Reality

Today in Scientopia, I discuss views of research time and money in the context of grants:

Do you stick to your plan as closely as possible, or do you take a more flexible approach depending on how the research evolves? I tend toward the latter, but am working with some people who prefer the former. Can this collaboration be saved?

Possibly Well Written

How important is writing quality for the success of a grant proposal? I don't think you can get a grant proposal funded just because it is well written, but of course it helps if you can explain clearly what you want to do, why you want to do it, and how you are going to do it.

For reviewers, it's good if a proposal is written well enough that it isn't annoying to read. A proposal filled with typos and 2-page long paragraphs consisting of a multitude of unrelated points is a chore to read and makes you wonder whether the poor writing reflects something significant about how the research would be done, even if you know that there may or may not be any correlation.

I have found that it is fairly common for reviewers of my proposals to comment on the writing of the proposal. When I am reviewing a proposal, however, I tend to comment only if the proposal is extraordinarily difficult to read owing to writing problems; that is, the writing is so bad that I am not really sure what the PIs are trying to propose.

If a proposal is well written or moderately well written or not especially well written but I can still figure out what is going on, I don't tend to comment on the writing unless I can think of some specific constructive comment that might be helpful (e.g., for a new investigator). I comment on the writing of manuscripts submitted to journals, but what is relevant in a proposal review is different from what is relevant in a manuscript review.

I was thinking about this because I recently read the reviews of one of my proposals, and I noticed that 3 of 6 reviewers commented on the writing of the proposal:

The proposal is well written..

This is a well written and prepared proposal..

The proposal is not particularly well written..

Since neither of the positive comments about the writing said that the proposal was very well written, and the negative comment used the somewhat feeble description "not particularly", I am going to conclude that the writing was OK -- not great, but good enough. From the rest of the comments in those reviews, it seems that the first two liked the overall proposal anyway, and the third one found lots of little things to criticize -- nothing fatal (the grant was funded), but the reviewer clearly had some other ideas about how the research should be done. In that case, "not particularly well written" might mean "I would have written this proposal in a different way".

In another recent proposal that also led to a grant, two reviewers commented on the writing:

This proposal is very well written..

This proposal is well written..

OK, that's nice, but not relevant unless the reviewers took this into account in their overall ranking. There's no way to know if they did; see below for question about this.

But first, in the interests of bloggy pseudo-research, I need to do something unpleasant and re-read the reviews of a proposal that did not lead to a grant.. a rejected proposal. What, if anything, did reviewers say about the writing in my failed proposal?

Only one out of 6 reviewers mentioned anything about writing:

The proposal is well organized and well written..

Of course it is not possible to conclude anything from these few examples. The reviewers were likely different for each proposal, and who knows whether these reviewers make a habit of commenting on the writing.

I nevertheless stand by my rather obvious hypothesis, expressed in the first sentence of this post, that good writing won't get you a grant (if the proposed research isn't Excellent).

This leads me to some questions for readers who review proposals:

Do you factor how well written a proposal is (or isn't) into your overall proposal rating?

Do you typically mention the writing in your review? (always, never, only if the writing is notably good/bad?)

Do you think that good/bad writing could tip the scale for a proposal to be funded/not funded if the proposal is right on the very edge of the funding zone?

Dollar Thrills

We interrupt our regularly scheduled post to exult in what I hope is only a mildly obnoxious way about a new grant.

It's not as if I have grants raining from the sky all the time, but, in the course of my career, I have been fortunate to have had pretty good success/luck getting grants. So, although I certainly would never say that getting grants is routine for me, it's not a rare occurrence.

So this is what I was asked myself: After 15-20 years of getting grants, is it still exciting to get a new one or is it more like "OK, cool, here's another chunk of money to spend for a few more years"? Maybe it's the latter for people who have mega-labs with an endless stream of grants rolling in, but for me:

IT IS STILL VERY THRILLING.
IT IS DEFINITELY STILL VERY THRILLING.

In fact, when I get the good news about a new grant, I am always reminded of how exciting it was to get my first big grant many many years ago. Of course, the first big grant as an assistant professor came with an element of relief (as in, 'maybe I will actually get tenure' relief), but, even at mid-career, there is still a very similar feeling (as in, 'maybe I can actually support all these grad students and postdocs for a few more years' relief).

But the feeling is much more than just relief. There is also the satisfaction of the effort of proposal-writing coming to a successful result (against the odds). But mostly it's the thrill of being able to do extremely interesting research with an extremely excellent research team.

Then I asked myself: Does the magnitude of the thrill scale with the $ amount of the grant? That is, is it just as exciting to get a $40,000 exploratory grant as it is to get a $4,000,000 mega-grant? Yes.. but not quite as thrilling. I must confess: Really big grants are really thrilling.

From Proposal To Grant

In the good old days, this was a typical scenario for NSF grants, indicated by time (t) in months:

t = 0: proposal submitted on or before the proposal deadline. (I was just trying to remember if I have ever submitted a proposal before a deadline, and I don't think I have.)

t = 3.5-4: panel met, discussed mail reviews, ranked proposals

t = 4-4.5: program directors made decisions. Those PIs who were definitely funded and those who were definitely not funded got the news right away; those who might be funded waited a bit longer.

t = 5-6: proposals recommended for funding worked their way up through the system, but it didn't take long to get the final award letter and for the funds to be transferred to the university and for the university to assign the grant an ID number, making the grant active.

Back in those glorious days of yore, when filling in the cover page, I would indicate a preferred start date 6 months after the proposal due date, and this was quite reasonable. A new grant was typically good to go after t = 6 from time of proposal submission.

This was an excellent system because you knew that if your grant was funded, you could start paying a graduate student RA in the next academic term. This was very useful for making a quick start with the research, for trying to optimize correspondence of graduate students and grants, and for research planning in general (e.g., if you get/don't get a grant, this affects your plans for the next proposal deadline).

Now the system seems to be more like this:

t = 0: proposal submitted on or before the proposal deadline.

t = 3.5-4: panel meets, discusses mail reviews, ranks proposals

t = 4-8+: program director makes decisions, contacts PIs

t = 6-10+: proposals recommended for funding work their way up through the system, and eventually result in the formal award letter, which will be followed at some point by transfer of funds to the university.

And then.. there is another wait to get the university to assign a magic number to the grant so that it can be used. This delay is beyond the control of NSF, but it is one more delay on top of the other delays, making for a considerable gap in time from proposal submission to effective grant start date. I specify effective grant start date because now the start date indicated by NSF may be a couple of months before the university completes its paper work and recognizes the grant in its grants management system. The grant may technically exist for a while before the PI can use it.

It is sometimes possible to start using a grant that has been promised but that hasn't quite worked its way through the system to the final stage yet, but to do this you have to have a slush fund (not another NSF grant) as back-up.

Why has this all become more complicated and slow? Is it because budgets have been slashed and there are fewer staff handling more responsibilities? I have a hypothesis, but it actually involves the opposite of this explanation.

I know that there are all sorts of considerations involving oversight of every step of the system and oversight of the overseers and so on and this all takes time to make sure no one is doing anything unethical or illegal of unwise, but I wish we could go back to the 6 month proposal-to-grant time gap. Gaps that are considerably longer than that seriously interfere with a PI's ability to assemble an excellent research team and do exciting research in a timely way.

I am of course happy to get grants at all, so it might seem ungrateful to complain about a delay of a few months, but I have found that the delay has rather significantly affected my research program.

Not For Your Information

A reader writes with this cautionary tale:

I don't know if you've heard about NSF FOIA requests, but you can make
an FOIA request to get copies of funded proposals from NSF. A
colleague of mine told me about this; that all NSF proposals are
public and that's why they have FOIA requests. It certainly sounded to
me as if NSF has a public repository or library of proposals where
anyone could easily get access. It also made sense to me --- research
papers are public, so why not funded proposals? So I made a request
for a number of proposals that looked interesting and relevant to me.

However, it turns out that it's not the case at all. NSF folks
themselves don't seem to like FOIA requests and they contact the PIs
to get approval. Then there's paper work involved. And so on. But a
bigger problem is that some people get offended by this request.

So after I found this out, I withdrew the request and sent out emails
to numerous PIs to apologize. And I will certainly tell all of my
colleagues to never do this.

I can certainly see why some PIs would be freaked out that someone was requesting to see their proposals. Proposals contain our unpublished ideas and plans. We already have to trust reviewers and panel members with these ideas, and hope that no one will steal them before we have a chance to carry out the research and publish the results. I do understand that these are public documents in some ways, but they are also very sensitive documents, and shouldn't be immediately available to anyone who wants to use or misuse them.

Of course there are innocent reasons why someone would want to see a proposal, particularly an early career faculty member who is curious about the research topics in certain proposals and who might benefit from seeing how others put proposals together. There are, however, less perilous ways to achieve some of those same goals; for example, asking senior colleagues or mentors if you can see their proposals, working on new proposals with these same colleagues, serving on a proposal review panel, communicating directly with PIs about mutual research interests, and/or inviting researchers of interest to give a talk in your own department.

I am glad that my correspondent wrote to the PIs of the proposals he requested to apologize and make it clear why he requested their proposals. This was a good thing to do. If I were one of the PIs, I would not hold it against the young professor who was just curious to see some interesting and possibly useful proposals.

And I agree that it's not a good idea to make a Freedom of Information Act request for someone's NSF proposal, even if you can.

Grant Expiration

Today in Scientopia, I discuss the mismatch between the time frame of a grant and the time frame of graduate students.

The Odd Women

A scientist put together a research team to do some Awesome Science, and they wrote a grant proposal together. The team involved three PIs at different universities, their graduate students, postdocs, undergrad research students, and so on: the usual elements of a research proposal. The proposed research was excellent, and all 3 PIs are talented, respected scientists. Even so, some people doubted whether the proposal would be funded.

There are lots of reasons to doubt whether any particular proposal, however excellent, will be funded, but in this case, there was a reason unrelated to the significance of the proposed work, the research qualifications of the research team, the technical merits of the proposal, or even the lack of enough money to fund all excellent grant proposals. So what was the reason for doubt in this case?

The 3 PIs are women.

The doubt had nothing to do with the qualifications of any one of the PIs (all are highly qualified), but rested on the fact that all 3 PIs are female.

Of course, in the entire history of humankind, including today, no one would blink an eye if there were 3 male PIs. In the physical sciences, most scientists are men, and this has always been the case, so an all-male team of PIs is unremarkable, even today.

Oh sure, nowadays male PIs might describe in a proposal how they would involve female students or postdocs in the research, and they might write about how some of them have even advised females in the past, but the proposal would be funded or not funded depending on the scientific merit of the proposed work. I see proposals like this all the time.

But, even considering that this field of science is dominated by men, especially at research universities, is it so strange that a research project might be led by three women?

And, even if it is unusual, is it a problem?

In fact, it was not a problem. The pessimists were wrong, and the proposal was funded. And it was funded because the science was great and the PIs are all leaders in their field, with substantial track records of excellence and productivity.

Now, some would think the proposal was funded because the 3 PIs are women.

Are you following along? It's confusing, I know, but here is a handy summary:

If the proposal is not funded, it might be because the 3 PIs are women.
If the proposal is funded, it might be because the 3 PIs are women.

Fortunately(?), these statements refer to perceptions, not reality. Today, projects led entirely by women, even in STEM fields in which women are vastly underrepresented, are funded by the NSF if the proposed research is excellent.

We are not yet, however, at the stage where an all-female team of PIs is unremarkable. We are still at the stage where some people wonder if women get grants because grant agencies have to fund some females to meet diversity quotas or worry about projects that involve too many women. Maybe we shouldn't worry about this. Maybe we should just be happy (for now) to get grants. Maybe it is asking too much to have grants and respect?

This started me thinking about what an unremarkable proportion of women would be on a project. Is it equal to or less than the proportion of women in a particular field, or can we crank that number up a bit? In a 3-person PI team in a field in which women represent much less than 1/3 of scientists at research universities, we'd have to round a fractional woman down to zero if we used the proportional scheme.

I'm going to propose that, in fields such as this, ~25 +/- 10 % would be a non-threatening, unremarkable % of women involved in a research team. At this level, most people wouldn't worry that the science won't be any good or that talented men are being excluded for unfair reasons that have nothing to do with their research skills or experience.

But actually, my preference would be to assume that my not-entirely-serious, cynico-sarcastic analysis is flawed -- the proposal with the 3 women PIs was funded, after all -- and let some people go ahead and worry that maybe it was funded because NSF has to toss some money to the girls now and then.

Eventually (soon?), anyone who remarks in a negative way on the oddness of an all-woman PI team will be told that they, in fact, are the odd ones to think there is anything remarkable or problematic about an excellent and productive research team that just happens to involve only women.

Note: The title of this post is an oblique reference to the George Gissing novel, The Odd Women, in which "odd" doesn't mean that the women are bizarre, but refers instead to the fact that, in late 19th century England, there were more women than men of marriageable age; i.e., an "odd" number of women. The title of this post therefore refers to a possibly problematic circumstance in which there are too many women.

Misadventure

There has been much blogospheric discussion, here and elsewhere, of (good) ways in which "career interruptions" can be mentioned in grant proposals, so that no one is penalized for a temporary decrease/halt in productivity owing to certain important life experiences (babies, illness, elder-care etc.).

My personal favorite way in which information about Personal Interruptions is requested is in the instructions for writing Australian Research Council grant proposals:

F14.1. Provide and explain:
..
(iv) Any career interruptions you have had for childbirth, carer’s responsibility, misadventure, or debilitating illness;

It's great that this is included in the proposal instructions, and I don't mean to make light of an important issue that has only recently been adopted by large federal funding agencies in certain countries, but I must admit that the request for an explanation of any "misadventure" is rather intriguing.

In fact, I don't think I can wait until December for my usual (northern hemisphere) winter break FSP contest on some (strange) text/document related to academic life. So here is the challenge:

Provide and explain any career interruptions that you have had for misadventure, real or (better) imaginary. Your explanation cannot exceed 475 characters (with spaces). For example:

I was unable to submit any articles for publication between 23 August 2007 and 13 November 2008 because I was kidnapped by pirates and, although I was not otherwise mistreated, I was not allowed access to the Internet. I did, however, scratch out some manuscript drafts on spare pieces of sailcloth using a gull feather and an ink mixture that I made from mussel shells soaked in beer, so as soon as I was released and had Internet access, I was able to resume publishing.

or

Immediately upon receiving tenure in 2005, I was beamed aboard a spaceship on a secret mission I cannot reveal here. At first I was unable to communicate with the life-forms piloting the craft, but over the years I learned their language, customs, and the songs they like to sing on long journeys. Eventually they returned me to my office, and I have subsequently resumed my academic career, no worse for wear but with an unfortunate gap in my CV.

Now it's your turn to describe your career-interrupting misadventures.

Orders of Magnitude

Thanks to all who sent me links to the recent PNAS article by Ceci & Williams on "Understanding current causes of women's underrepresentation in science". I had read reports of the results, and now have read the article itself.

The authors of the article focused on some of the most commonly cited reasons for the underrepresentation of women in "math-intensive fields of science": discrimination in reviewing of proposals and manuscripts; being selected for interviews in faculty positions; and in hiring. They propose that current data do not support these as primary causes for the underrepresentation of women today.

Their main conclusion is that "differential gendered outcomes in the real world result from differences in resources attributable to choices, whether free or constrained", and that underrepresentation would be best alleviated through changes that take into account "differing biological realities of the sexes."

I think they make some important points with their study, and I believe that the current situation for women being evaluated for jobs, grants, and publications is better than it has ever been. However, I continue to see and hear examples of discrimination in reviews and hiring committees -- faculty who doubt that women have their own ideas (but have no trouble believing this about male candidates), or who don't like "aggressive" women (but think this is a fine trait in a man), are alive and well. These issues are not as obvious and widespread as in the past, but neither are they isolated, rare incidents that can safely be ignored as irrelevant to current practices.

I was surprised that the study focused so much on databases related to the life sciences, a realm of science in which, as the authors note, women now make up the majority of PhD recipients. I realize that some biological sciences are quite "math-intensive", but the authors seem to use this term to refer specifically to the non-biological sciences. It would therefore make sense to base conclusions primarily on studies other than those involving the life sciences.

(A quibble: In the sentence about how the number of PhDs awarded to women in the life sciences has increased, 13% is described as "orders of magnitude less" than 52%. It is not.)

In the section on Discrimination Against Women in Journal Reviewing, the authors rely in part on a study of acceptance rates for the journals Behavioral Ecology, the Journal of Biogeography, and Nature Neuroscience. Those all seem kind of life sciencey to me.

Similarly, in the section on Discrimination Against Women in Grant Funding, the authors rely on studies of databases of Medical Research Councils and similar organizations of various countries, including the NIH. There is also mention of NSF and the Australian Research Council, both of which cover a wide range of fields in science, engineering, education, and beyond. If possible, it would have been interesting to see a separate analysis of recent data for the "math-intensive" sciences.

Having seen such data for my own field, I believe that NSF works -- with some success -- to provide a "gender-fair grant review process", but I don't think the authors of this particular study have demonstrated that with their chosen databases. [I chose the phrase "works to provide" rather than a simple "provides" based on experiences such as this (which referred to, but did not specify, that the problem was offensive sexist comments by a program officer) and this and this and this, all of which the excellent and enlightened staff at NSF can and do filter so that there are no deleterious effects on female PIs].

The authors note that there are "more women in teaching-intensive, part-time posts where research resources are scarce", and attribute this to life-style choices or constraints. When discussing the situation at major research universities, the authors cite an NRC task force report that concluded that women were not at a disadvantage for interviews and offers (and may have a slight advantage) in a study of 6 fields of "natural science". That's great, but I think it is premature to propose, based on these data, that universities should discontinue efforts to train hiring committees to avoid bias (explicit or implicit).

Although ultimately not as convincing as it could be (owing to the datasets used), this is a useful study in that the authors try to focus our collective attention on additional factors that affect the underrepresentation of women in math-intensive fields of science, and suggest that universities explore new options for career tracks ("The linear career path of the modal [sic?] male scientist of the past may not be the only route to success.."). I agree; just don't throw out the methods that seem to be working so far. That would make the situation orders of magnitude worse than it already is for women in math-intensive fields of science.

Proposal Yin and Yang

A few years ago, some colleagues and I pitched a 'high-risk' idea to program directors at a funding agency. The program directors were receptive, but funds were scarce (of course) and they suggested that we submit a full proposal through the usual channels, get destroyed in review, and then they could use the short-sighted negative review comments to support a case for circumventing peer and panel review.

Well, they didn't describe it exactly like that, and they did hold out the possibility that our pessimism about taking this idea through normal review channels might be unfounded -- perhaps we could get the full proposal funded on the first try!

So we wrote the proposal and were pretty thoroughly destroyed in review. Reviewers were convinced our central idea was misguided, even absurd. (spoiler alert: They were wrong.).

But we got a pilot project funded, and gave the project a try. I was convinced that the research was worth doing, but I thought it would be more difficult than it turned out to be (smug alert: I was wrong). In fact, within 6 months we had the results we needed to validate the idea and show that it was worthwhile to move forward with this line of research.

I don't really blame the skeptical reviewers. Although sometimes it seems like PIs need to do most (or all) of a research project before we can convince reviewers and panels that the project is doable (and worth doing), I admit that in this case, the idea was a bit surprising.

Nevertheless, armed with our new and exciting data, we started writing a full proposal again. The new proposal was even more fun to (re)write than the first proposal because we had the new preliminary data as strong and (I think) compelling core of the proposal. As the proposal-writing progressed, I kept reminding myself to look over the old reviews in case there was something worthwhile in there that should be dealt with in the proposal rewrite. Even in situations where reviews are overall not very useful, there might be something useful that can be mined from them.

For example, there might be some minor points that need attention. These might be minor, but it's a good idea to take these into consideration in case the proposal goes back to the same reviewers.

But I kept putting it off. At pretty much the last minute, I skimmed the reviews of the old, rejected proposal.

I am not usually such a coward about reading or re-reading negative reviews, and I didn't really feel particularly wounded by the reviews, even the first time I read them. I just felt like it would be a real downer to read those reviews again, and I was in a very good mood about the new proposal. Yes, I know we might be eviscerated in review again, but at the time of (re)writing, I was in that happy delusional state when I really like the proposal and am excited about the prospect of doing the project. I was reluctant to put a dent in that happy mood. And I knew that one of my colleagues had re-read the reviews in detail.

I mentioned to another colleague -- one who is not involved in this particular project -- that I needed to re-read some negative reviews of a proposal I was rewriting, and he said that he would rather have his fingernails ripped out than re-read old reviews of a rejected proposal.

That seems a bit extreme, but sort of captures the essence of my feelings about the matter.

I am not advising that anyone not re-read negative reviews when it is clearly in one's best interests to do so. It must be done, however unpleasant, but I think it is also important to find a way to minimize any damage such a (re)read might do to the positive energy that is driving you as you revise a proposal (or manuscript).

It occurred to me that, had I been thinking ahead when I read the reviews the first time, I could have jotted down any points worth addressing in a future proposal, and thereby avoided the full re-read later.

But if, like me, you don't think about that in time and you have to re-read the stupid reviews, perhaps the re-reading is best done in a pleasant cafe (or bar), or at home with your most sympathetic pet nearby, with your favorite music playing. Or you could re-read the annoying comments on a crowded bus or subway or at the dentist; that is, some place with unpleasant distractions. Or read them with a good colleague, take whatever is useful from them, and laugh about the rest.

My advice: Read them, but don't let them make you defensive (in your proposal) and don't let them stop you from enjoying writing your cool new proposal.

Art & Si

As longtime readers know, every once in a while I venture out of Science World and interact with colleagues who inhabit different parts of the academic planet. Most of these interactions are very interesting, educational (for me), and fun. Some are disconcerting. Like this one:

During a meeting to discuss research funding, including for students, a professor from a discipline in which grants are rare thought that a professor in a rather more grant-rich field had put together an unrealistic budget for research involving a grad student. Let's call the first professor "Art", and a committee member, from a similar field as the professor who constructed the budget, "Si".

Art, pointing to the budget line for the student's salary, said: Look at this! The budget has a month of salary for the grad student to do field work in the summer!

Si: Yes.. that looks reasonable.

Art: What?!?! Why does the student need to be paid? It's for field work, so presumably the professor is paying for the student's food and travel and whatever. The field work is even for the student's own thesis research, so the student has to go on the trip. Why does the student need to be paid?

Si: Umm... because the student will be working and the student is not a slave?

It was big news to Art that many of us pay our graduate students to do research in the summer: to work in the field or in the lab or in an office in front of a computer. It somehow seemed excessive to Art that students would get their travel paid AND also get a salary while doing research that benefited their own thesis.

Art thought that we paid our students a research assistantship only for research that is unrelated to their thesis research. The scientists and engineers on the panel were stunned that anyone would think that.

Some of the things we learn from each other on these multi-disciplinary panels make us all feel good, as if our intellectual boundaries have been stretched. This was not one of those times. I felt strangely sad that it was news to Art that we pay our students to do thesis research. I was glad that faculty summer salary from grants was not an issue in our deliberations.

The Long Haul

Some research projects take a long time to get off the ground, if they ever even make it, and it can be difficult to keep everyone in a research group focused and motivated during the long (2-3+ year) time required to determine if some projects will be funded.

One project with which I am involved is so far doing OK in the protracted process required for this type of large project, but it is too early to tell if it will ever be funded. We survived a pre-proposal and one round of proposal review, and are gearing up for another round in the coming year. These types of large proposals are never funded the first year, so I expected this to be a long and arduous process.

The research group is quite large and multi/inter-disciplinary, and is also very international. At the outset, I told everyone in the group that the proposal would not be funded the first time, and it would take a minimum of 2 years to get it funded, and of course there is always a (good) change it might never funded.

The US researchers were all familiar with this and accepted the reality of the long march to (uncertain) funding. What I didn't expect, however, was that some of our international colleagues didn't really believe that a rejected proposal would ever be funded.

One colleague who is in the UK system told me that he can only submit a proposal for a particular project once; if the project isn't funded after one round of review, the project is dead. He therefore viewed any proposal not funded the first time as somehow damaged and unlikely to get funding. A few other researchers in other countries were similarly skeptical, and were unsure whether it was worth their time to continue as part of the research team.

I can understand their skepticism. This system is not very efficient and it's hard to see a rejection of a proposal as a step forward, but rejection is commonly part of the funding process. I think that part of the philosophy behind the protracted process, especially for large, multi-interdisciplinary proposals, is to maximize the chances that the research teams will function well and optimize the use of grant money. The proposal with which I am involved has definitely benefited from the various review stages.

At some point, though, the length of time to get (possibly) funded has a negative effect on the timeliness of the research objectives and on the ability of the research team to maintain cohesion. Furthermore, it is difficult to write the project to involve specific postdocs or graduate students when the time frame of the process is so protracted. For these reasons, the long time frame of the review process decreases that chances that the research will be as good as it would have been if funded the first time.

In the meantime, I am trying to keep a large research team together and keep the proposal process alive for a bit longer because I think it will be worth it to try again. Beyond that, however, I'm not sure it will be worth it, even though some groups get funded after 3+ years of effort. In this case, however, I think the research ideas wouldn't be very fresh after that amount of time, and it would be better to try something else.

Of course I am hoping that I won't have to make that decision. At least for me, delusion is essential to the proposal-writing process.

Survey of Impact Criteria in Peer Review

Last week, an e-mail arrived from someone who is

"..part of a research team conducting a National Science Foundation-sponsored study that may be of interest to visitors to your website. It concerns the comparative analysis of the grant proposal peer review processes at six public research funding agencies (http://csid-capr.unt.edu/)."

The current phase of this group's research at the University of North Texas includes an online survey of Impact Criteria in Peer Review. I have been asked to help advertise this survey.

Click here to participate in the survey. It takes about 20 minutes to complete, maybe more if you spend a lot of time pondering the complexities of the questions and possible responses.

After establishing some things about you and your employment status, the survey asks you to indicate which funding agencies are relevant to your life, and in what capacity you have been involved in these agencies.

And then: In your opinion, who should be involved in the peer review process and decision-making? Peer academics, international academics, funding agency officials, other government officials, industry people, "lay" people of the US or other countries, those most affected by your research?

The weirdest part is a series of statements about the philosophy of research and its impact on society. You are supposed to say whether you think your funding agency's "orientation" agrees or disagrees with some statements, e.g. "There ARE NO SIGNIFICANT BENEFITS to society from public funding of scientific/technical research." Strongly agree? No opinion? Strongly disagree? (and some intermediate choices)

Technology happens, society must adapt. Agree.. or not?

And then answer those same questions for what you think the LAY PUBLIC thinks. Is this a measure of how cynical we are?

Another question deals with how we think GOVERNMENT OFFICIALS (and others) value "instrumental" vs. "intrinsic" research.

And yet another: Can the intrinsic value of research be measured? I suppose everything can be measured, but does the measurement mean anything?

That's Part 1, which the survey authors admit is a bit "abstract".

In Part 2, which does not need to be completed, survey responders give their opinions (on a scale of 1-100) about some inventions selected in 2009 by Time Magazine as the most important ones of the previous year. You also guess what the average answer was from "the public" who read the Time article and responded online with their votes. You can see how your guesses correspond to the data by requesting the results via e-mail.

I did not find the survey particularly illuminating, but I wonder what will be done with the results of this survey and the evaluation of the results.

Feel free to leave additional comments here on the FSP blog if you have thoughts and ideas to share about the survey or the issues it seeks to evaluate. Is this a good way to determine what researchers think the public thinks about federally funded research? What should be done with such data?

CAREEReview

Owing to the apparent youth of much of the academic segment of the blogosphere, it is not difficult to find posts by assistant professors describing their activities and thoughts during various stages of preparation of CAREER proposals, and their elation/dejection upon receiving news of NSF's decision.

My own CAREER grant expired long (long) ago, and now my primary experience with the CAREER program is as a reviewer of proposals and as an occasional mentor (in real life) to colleagues who are preparing their own such proposals.

Fellow reviewers of CAREER proposals (or whatever equivalent there might be in the NIH university): let's discuss how we go about reviewing these things.

If you have reviewed one or more CAREER proposals, do you:

1. Ignore the fact that these proposals are a bit different from regular NSF proposals, even though you know better, and review them much as you would any proposal (perhaps accounting somewhat for the relative career youth of the PI). That is, do you mostly ignore all the extra educational components of the proposal and focus on the Science?

2. Pay some attention to the education and/or outreach plan, but focus mostly on the Science?

3. Give serious (perhaps equal) thought to the plan for educational/outreach activities, provide detailed comments on these in your review, and factor in your opinion of these in your overall rating of the proposal?

- Other..?

From what I've seen through various intersections with the CAREER program over the years, different fields have different philosophies about these grants and how important they are. I think everyone agrees that it is a good thing to get these grants, owing in part to their being of longer duration (up to 5 years) than a typical grant (2-3 years). There seem, however, to be differences in practice about when to submit the proposal, e.g.:

The CAREER grant can/should be the first grant obtained by an assistant professor, and therefore should be applied for early, even if the individual has no previous grant track record,

vs.

A CAREER proposal should be submitted after there is some grant track record, even if after only one other NSF grant.

And I have heard rumors of fields or subfields in which a CAREER grant is essential for tenure at some institutions, but I have not seen evidence that these rumors apply to anyone I know or to any department at my university as long as there is otherwise a solid record of funding.

When I review a CAREER proposal, I certainly look at the required education/outreach parts and I like to see a sincere effort with this part of the proposal, but I must admit that I think it is asking a bit much of assistant professors to have a particularly sophisticated plan. It's great when someone really does have a creative and detailed plan, but, as long as there seems to have been a sincere effort with this part of the proposal, I don't penalize PIs whose broader impacts aren't awesome. That doesn't mean I don't value education or outreach; it just means that I think we have to be reasonable about expectations for already overburdened early-career faculty at research universities.

So, I guess I'm mostly a #2 in the list above, with the caveat that the education/outreach plan has to have some substance to it. After all, that is part of what distinguishes this grant from others, and partly what justifies the longer duration.

Another issue arises when reviewing CAREER proposals from faculty at different types of institutions. This is a general issue when reviewing proposals from, say, faculty at research universities vs. faculty at small liberal arts colleges, but it's a particular issue for CAREER proposals because the education activities might be more of an expected and valued part of the job for some faculty than for others. I actually don't think it's a major big-deal issue because we routinely deal with these types of differences n reviewing proposals, but perhaps someone disagrees with that opinion?

So, how do you review CAREER proposals?